[WIP] layers: list: organize languages

[Anton-Latukha]

List of language layers in Spacemacs is huge. Pull request organizes languages.

Reason to do so came from personal experience of configuration. List was long enumeration of everything and their uncle. As DevOps needs to look in many types of files/data, I needed to look several times through it to be sure I picked all that is helpful to me.

Also there is a factor of: "Interesting, what that language is for" and "I forgot what that language is for".

Categorization

Yes, there are a huge number of ways about how to categorize programming languages.

I thought about classifications of this list, and from time to time studied classification of languages, for a half of the year.

And this is the conclusion.

General-purpose OR Domain-specific

There are two major defined domains of computer languages:

• General-purpose programming languages (system programming, general-purpose scripting languages included)
• Domain-specific languages

Paradigm classification

All programming languages by themselves have solid paradigm classification.

Additional points

Language paradigm choices are targeted to fulfill the areas of application of the language. So areas of application that language created for has an influence on design of language and paradigms chosen. That is why area of application is primary to the paradigm.

Language can be purely functional, and have very narrow domain-specific application (Coq, Faust), making it not possible/useful in other areas.

Many of domain-specific languages are not Turing-complete (programming in the strict sense) languages (example - markup languages). That is other point why area (target) of application is primary to the paradigm.

Many languages are multi-paradigm, due to they are had been multi-paradigm from the start design, or because of the imperative -> functional paradigm creep, or due language having not purely functional design, that enables it to have other paradigms.

To move language from multi-paradigm to imperative or other group we need to have a field professional statement, that the language possibilities in some paradigms are very rudimentary, and that language is primary belongs to one paradigm. Otherwise language belongs to multi-paradigm.

Many general-purpose languages initially were/are scripting languages, but most of general-purpose scripting languages are Turing-complete and offer general-purpose grade programming libraries/tooling, and also now have compilation possibilities, and other traits of classical general-purpose languages. That is why general-purpose scripting languages group is not formed, because it is hard to strictly state they are scripting - they are general-purpose languages.

Major C family group is also hard to form. It is OR half of languages in broad sense, OR (c-c++ & d) in strictest sense, that makes a group too small (only two layers).

Where strict sub-goups of more then two by the main trait could be created - they was grouped (JavaScript (since purpose of TypeScript and PureScript is to produce good JavaScript output), Lisp dialects (and link to Emacs Lisp being DSL), DSL -> {Markup, {Windows, Unix} Scripring}).

Also there are a couple of explanations, so mortal-ones can instantly see that ess is a bundle for R, and what is in gpu layer.

I am open to make additional touches in the name of the strict sense.

[syl20bnr]

Great work! I would like to see this in Spacemacs but it needs more work on the Spacemacs side because the layers.org file is generated automatically by a script from the file tree!

So this great work will be overwritten when the file is regenerated :-(

I'm not fond of moving all the files physically to get this classification but I like the fact that the layers.org file is generated, so we need a logical solution to be able to do both.

A first solution that comes to mind is to define a new conventional variable xxx-sub-categories which takes a list of sub-categories from root to leaf:

(setq clojure-sub-categories '("Multi-paradigm" "List dialects"))

[syl20bnr]

Instead of xxx-sub-categories let's call it xxx-categories and leave the physical organization of the layer completely independent from it.

So it becomes:

(setq clojure-categories '("Programming Languages" "Multi-paradigm" "List dialects"))

Maybe in the event where there is no xxx-categories for a layer we can fallback to the physical organization.

[Anton-Latukha]

1. In Programming languages Markup languages sub-entries reduced the words in names (aka removed ... languages.

2. Added a optional commit that has alphabetical org sort check, all it did holds - swapped Emacs and E-mail.

[Anton-Latukha]

I wanted to add:

• I have a epic pull request, POSIX Nix installer. That one is going for a lo-ong time, and it is a hard one (a huge pile of shell script that needs to be very flexible and solid - which by itself is oxymoron), hard to test and hard to merge that one. And I want to get that done ASAP. NixOS 18.09 approaching. And I want Nix installer to be tested by community before 18.09 hits the news. So when people going to try Nix package manager in all Linux/Unix systems - it going to install presently and work perfectly, to create best first impression.

• I also just learning org.

• And I not written Emacs List for a long time and forgot all basics I knew. So I need to refresh Lisp to make anything thoughtful with it.

So I have a mix, in my org skills I am not sure, while also Lisp I need to learn, and that is also why I have a trouble understanding how to do the Lisp/org right here.

So I would not mind:

if (someone would take everything from me further)
  then - it should be trivial to finish, for someone who speaks Lisp already
  else - I would return after Nix installer is at least in "Is waiting for merge" state, and would make last swing at this.

[syl20bnr]

@JAremko is working on improving the categorization of layers and auto-generation of layers list. We may add tags via the org file of the layer.

[Anton-Latukha]

@JAremko

Good day.

This PR in brief - categorization of program language layers by language families and paradigms.

Above comment reference your work on layers lists infrastructure.

I have an entry ELisp skill, and knowledge in functional programming.
Please, if you can direct me with meaningful lower-level instructions (AKA in terms of lists and functions), how layers list generation works now, then I would be able to make this change.

Thank you.

[JAremko]

@Anton-Latukha Most of documentation processing written in Clojure and resides in the separate repo: https://github.com/JAremko/spacetools
l'm planning to rewrite legacy LAYERS.org generation this month then we'll have a way to make the structure that you created persistent. Currently it is second in my TODO list 😉

[Anton-Latukha]

Ok, thank you. Then my stall wasn't in vain =)

If you would deside on the fly - feel free to implement this.

If you would leave this for me - I would ping you in 2 month for general directions.

Thank you.

[JAremko]

@Anton-Latukha I added new layer generation system (it's pretty much WIP)

It is driven by tags of README.org files:

#+TITLE: Forth layer

#+TAGS: general|imperative|lang|layer

* Table of Contents                     :TOC_4_gh:noexport:
- [[#description][Description]]
  - [[#features][Features:]]
- [[#install][Install]]
- [[#key-bindings][Key bindings]]

* Description
This layer adds basic support for the Forth family of languages to spacemacs.
...

You can try generating layers file locally with:

docker run --rm \
  -v <SPACEMACS_REPO_ROOT>:/tmp/docs \
  -v <PATH_TO_CONFIG_FILE>:/opt/spacetools/spacedoc-cfg.edn \
  jare/spacetools docfmt /tmp/docs/

Config file looks like this:

{:spacetools.spacedoc.config/layers-org-query
 [{"lang" [
           "pure" {"general" [
                              "imperative"
                              "multi-paradigm"
                              ]
                   }
           ]
   }
  "markup"
  ]

 :spacetools.spacedoc.config/valid-tags
 {"lang" "Programming languages"
  "js" "JavaScript"
  "dsl" "Domain-specific"
  "e-mail" "E-mail"
  "distribution" "Distributions"
  "os" "Operating systems"
  "chat" "Chats"
  "misc" "Misc"
  "framework" "Frameworks"
  "markup" "Markup languages"
  "tagging" "Tags"
  "i18n" "internationalization"
  "pairing" "Pair programming"
  "web service" "Web services"
  "versioning" "Source control"
  "fun" "Fun"
  "script" "Scripting"
  "imperative" "Imperative"
  "general" "General-purpose"
  "pure" "Purely functional"
  "emacs" "Emacs"
  "vim" "Vim"
  "completion" "Completion"
  "tool" "Tools"
  "extra" "Extra"
  "lisp" "Lisp dialects"
  "layer" "layers"
  "theme" "Themes"
  "multi-paradigm" "Multi-paradigm"
  "checker" "Checkers"
  }}

:spacetools.spacedoc.config/valid-tags specifies what tags are allowed (keys of the map) and what names(values of the map) they'll have in LAYERS_WIP.org file.

:spacetools.spacedoc.config/layers-org-query is a sort of a database like query:

Here maps {<KEY> <VAL>} specify a group of layers/groups (will become a headline in LAYERS_WIP.org file). Vectors [ A B C D ... ] populate the group. Vector can contain another map (group) or a string. If it's a string then all layers satisfying full path query will be pooled in its place.

For example query like this ["lang"] will simply pull all layers that have lang tag.

---

Important notes:

• You can query for the same REDME.org files as many times as you want. So you can provide alternative groupings.
• Tags are case sensitive.
• Tags can have spaces but not | character (the're used to separate tags in README.org files)
• String values in :spacetools.spacedoc.config/layers-org-query are tags and must be present in :spacetools.spacedoc.config/valid-tags
• Empty LAYERS_WIP.org will cause an error. It will be empty if no query is satisfied.
• layer tag is implicit in :spacetools.spacedoc.config/layers-org-query as its root. So only README.org files that have the tag will be considered but the tool will try to guess (based on directory tree) and add layer tag if README.org has #+TAGS
• Not sure if it's buggy but I'll be back in a couple of days and hopefully will have time next week to fix them. 🐛 🔨

[JAremko]

@Anton-Latukha feel free to edit https://github.com/syl20bnr/spacemacs/blob/develop/.ci/spacedoc-cfg.edn and add tags to README.org files. Bot will update https://github.com/syl20bnr/spacemacs/blob/develop/layers/LAYERS_WIP.org automatically.

[JAremko]

Also this:

 [{"lang" [
           "pure" {"general" [
                              "imperative"
                              "multi-paradigm"
                              ]
                   }
           ]
   }
  "markup"
  ]

should be formatted like this:

[{"lang" [
          "pure"
          {"general" [
                      "imperative"
                      "multi-paradigm"
                      ]
           }
          ]
  }
 "markup"
 ]

I missformatted it.

[JAremko]

@Anton-Latukha I added almost all layers #12254 . If you want, you can take care of languages 😉

[Anton-Latukha]

Ok.
All seems understandable.
I would attend to understand how multiple ways of grouping you say gets solved. Or would understand what you ment.
I am annotating your explanation and can say that currently trying to make changes.
I would work today and tomorrow, and bring changes or report.

[JAremko]

@Anton-Latukha This evening I'll add "layer" and "uncategorized" tags to all layers that need to be tagged. So you'll just have to remove "uncategorized" and add proper tags to the README.org files.

[Anton-Latukha]

@JAremko
Understood and observed everything I needed.
And started working.

Expliit layer is a good idea.
Would continue after you merge uncategorized.

[JAremko]

@Anton-Latukha rebese your fork #12259

[JAremko]

@Anton-Latukha all that still needs a tag is under https://github.com/syl20bnr/spacemacs/blob/develop/layers/LAYERS_WIP.org#readmeorg-files-that-need-proper-tags

[JAremko]

@Anton-Latukha added new tags to the query and valid tags https://github.com/syl20bnr/spacemacs/pull/12260/files

[Anton-Latukha]

Currently, I pushed the 100% copy of structure that was published/discussed here previously, now in the new format. .ci/spacedoc-cfg.edn also adjusted to the structure.

[Anton-Latukha]

I am still thinking, maybe to include Markup languages as Programming languages/DSL/Markup languages, as theoretically they are such (aka computer languages).

[Anton-Latukha]

@JAremko
I also think it is a good moment to allow and start adding short descriptions near layers. Some layers that are a superset, like major-modes - obviously need attached description in the main list, to at least be discoverable.

[JAremko]

@Anton-Latukha Description headlines of the README.org files are copied into the corresponding LAYERS.org entries. You can add additional information into them (descriptions)

[Anton-Latukha]

The situation I observe swung my decision.
Included the Markup languages into the Domain-specific languages category.

They are really belong there by computer science nomenclature.
And since there are also jsonnet, semantic-web, yang, dhall they are blurring the line between simple markup languages, their templating, and programming, this languages demanded strict type hierarchy (place in DSL and Markup categories simultaneously).

I am making a local fork and doing manual testing now.

[Anton-Latukha]

@JAremko You are right. Docker stack really works great on local out-of-emacs fork. Now I see what you told about description being provided - awesome.

I observe these specific features of current document generation:

1. Main one: the targets in documentation exist only on the leaves of the structure tree. And not on branches.
   AKA:
   layers: lang: programming: general: multi-paradigm: js exists in documentation.
   All layers: lang: programming: general: multi-paradigm languages went into the black hole, and this I observe in all such cases, DSLs also. What would you say, how would we solve this situation. (my current structure config is posted below)
2. Tags get alphabetically sorted in README.md, with destroys their current visual logic continuty, but since they are tags and structure can change - I understand this is optional convinience.
3. Another big one: Layers in documentation do not get alphabetically sorted, good if they can be sorted.

[Anton-Latukha]

I would also include Frameworks as Programming languages/Frameworks.

[Anton-Latukha]

@JAremko
If something my current config block looks:

  {"programming" [{"dsl" ["lisp"
                          "markup"
                          "script"
                          ]}
                  {"general" ["imperative"
                              {"multi-paradigm" ["js"
                                                 ]}
                              "pure"
                              ]}
                  "framework"
                  "util"
                  ]}

[JAremko]

All layers: lang: programming: general: multi-paradigm languages went into the black hole, and this I observe in all such cases, DSLs also. What would you say, how would we solve this situation. (my current structure config is posted below)

@Anton-Latukha [{"dsl" ["lisp" "markup" "script"]} "dsl"] if you want to inject all "dsl" subtree there.
But it would be better if you add "general" or "misc" or mb some new tag to some of "dsl" layers to keep it all structured.

[Anton-Latukha]

@JAremko
Yes, the thing is - it lists there the whole subtree with subsets also.
I added a root tag.
Also instead we can use italic /*/ (aka *) to be more intuitive. In my expirience - it works good for Org files.

I submitted a current state.

The thing I noticed - since root tag pops-up several times in listing - Org result does not distinguish between:

  - [[#domain-specific-dsls][Domain-specific (DSLs)]]
    - [[#root][root]]
...
    - [[#multi-paradigm][Multi-paradigm]]
      - [[#root-1][root]]

And links all TOC root links to [[#root][root]].
So I think: or to solve it, or to supply unique tags.

[JAremko]

@Anton-Latukha TOC links seem to work properly:

https://github.com/syl20bnr/spacemacs/blob/1978fb207ba120372f0f79c4ce0993d893091bad/layers/LAYERS_WIP.org#root

https://github.com/syl20bnr/spacemacs/blob/1978fb207ba120372f0f79c4ce0993d893091bad/layers/LAYERS_WIP.org#root-1

---

Also instead we can use italic /*/ (aka *) to be more intuitive. In my expirience - it works good for Org files.

for what?

---

Yes, the thing is - it lists there the whole subtree with subsets also.
I added a root tag.

Hmm. I have couple of ideas how to make it less verbose. Hopefully without overcomplicating thing 🤔

[Anton-Latukha]

Ok. About links - was an empty message, I should thought more. Talked about org-links inside Emacs & it doesn't metter in what we do. Since web links work properly.

About Org view, since I do not know how it would be parsed in HTML site form - I'd would increase TOC to :TOC_5_gh: to show layers in multi-paradigm/root and multi-paradigm/js.

[Anton-Latukha]

@JAremko
So far so good. I am satisfied with the result. This chunk seems done.

[JAremko]

@Anton-Latukha I added layer sorting and now each README.org file can appear only once so root tags no longer needed - [{"dsl" ["lisp" "markup" "script"]} "dsl"] can be used instead.
Also READMEs that haven't been matched by the query tree will be displayed at the bottom of the layers file.

---

I'd would increase TOC to :TOC_5_gh: to show layers in multi-paradigm/root and multi-paradigm/js.

Yo can add :spacetools.spacedoc.config/org-toc-max-depth 5 to the configs (before the last })

[JAremko]

@Anton-Latukha Also this PR needs a squash into 1-3 commits.

[Anton-Latukha]

Checked TOC 4 -> 5 influence. It is minimal, across all Spacemacs doc files in total added several (~5-8) headers.

@JAremko - Done.

[JAremko]

@Anton-Latukha Are you sure that you used the latest image ? This seems like an old error https://circleci.com/gh/syl20bnr/spacemacs/32805?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link 🤔

[JAremko]

@Anton-Latukha I'll fix this https://circleci.com/gh/syl20bnr/spacemacs/32811?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link and we can merge 🤔

It happens because we have 2 identical headlines. I just need to merge them.

[Anton-Latukha]

@JAremko In current manner we collide with current CI validation rule "Multiply identical path IDs ... can happen if two headlines have child headlines with too similar names."
Conf form:

  "pairing"
  "reader"
  {"programming" [{"dsl" ["lisp"
                          "markup"
                          "script"
                          ]}
                  "dsl"
                  {"general" ["imperative"
                              {"multi-paradigm" ["js"
                                                 ]}
                              "multi-paradigm"
                              "pure"
                              ]}
                  "general"
                  "framework"
                  "util"
                  ]}
  "versioning"
  {"spacemacs" ["distribution"
                "util"
                ]}
  "tag"
  "theme"

[JAremko]

@Anton-Latukha Yeah I'll fix it by making the tool merge same headlines.

[JAremko]

@Anton-Latukha Should be fixed now (with the latest image) Hopefully last update 😛

Also don't forget to https://circleci.com/gh/syl20bnr/spacemacs/32826?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link 😉

[JAremko]

@Anton-Latukha I cherry picked it. You can delete the PR branch. 👍

[JAremko]

@Anton-Latukha Btw there still a lot some work: https://github.com/syl20bnr/spacemacs/blob/develop/layers/LAYERS.org#readmeorg-files-that-need-proper-tags