Module & example extraction?

[c-blake]

Someone was recently complaining to me about non-laser focused "only about CLIs" material in this repository. The rationale advanced was, I believe, a safety concern that import cligen/somesuch might compile when it "shouldn't" and pull in, well whatever is exported. While I politely asked him to create a discussion/issue here so that people besides just him could weigh in about backward compatibility desires, he didn't for whatever reasons. So, I am doing it instead. To me the main issues are discoverability, documentation, and ecosystem granularity.

With any modularity boundaries, there are always tough judgement calls - what names to export, etc. I tend toward "big tent" exporting. In my experience "chunkier" libraries help to sell platforms as well as reduce duplication. E.g., without scipy or Tensorflow which are both enormous, Python would not be used for much science. Even C++ might be nowhere today without the STL from the mid-90s (and I am no fan of STL). Meanwhile, on the other side, ripgrep, pijul or whatever Rust programs all seem to incur pulling in literally hundreds of dependencies which - at best - really stresses package management. I do not think Nim has a great story for package management. Diversity of package managers alone has been called "comical" to me in private correspondence.

I'm not sure there is any easy way to objectively measure "newbie frustration with an ecosystem" or if that is even the right "metric" or if there is even any single metric we could agree upon (the squeakiest wheels are often non-newbies). "Single module packages", much like single proc modules, disavow a modularity level and so seem more geared toward "selling" package managers. I also think some generic package name like "cbUt/" or "nimut/" or "nut/" is no more descriptive or discoverable than leaving things as they are. Some really love backward compatibility. So, beats me what the best ultimate solution is, but it seems worth discussing more broadly.

All that context/qualification having been made, a trend that began with examples/dups.nim back in April 2019 was to include modules that might just be generally useful for CLI tools on Linux/Unix even if not used internally. This includes (at least) the 17 modules m(file|slice).nim, dents.nim, statx.nim, (str|os|file|unix|posix)Ut.nim, tab.nim, prefetch.nim, the glob-abbreviation system (abbrev.nim, tern.nim, trie.nim), the magic.nim libmagic wrapper, conncomp.nim, print.nim as well as all of the 12 examples/ programs. So, 29./46 non-test modules are arguably "noise". Yes, yes..I called it a dump myself, but it's all under the cligen/ namespace and so not dangerous, merely untidy/amalgamated and 29 is not a large absolute number.

Lifting the examples/ into their own repo(s) would be easy since they are not really structured as an import or include anyway. So, this cannot really break backward compatibility and might afford better documentation.

However, removing things from the cligen/ import namespace would break backward compatibility of the library for "dark" (unpackaged/non-nimble) code. Due to its nature, there could be quite a lot of this for cligen. I know I have dozens of unpublished programs that use it (and various of these helper modules), myself. I can easily fix my own to point to new places, but I do not wish to antagonize other users with needless disruption.

So, we could have a (potentially long) deprecation period with pointers/URIs to new locations, and only do removal at some cligen-2.0 stage. I am open-minded about this and am also very open to suggestions, including e.g. "Not even 1 screenful of untidiness? C'mon, man! Do nothing!" If you do lean toward lifting out/separation for some cligen-2.0 then I would greatly appreciate any feedback on package boundaries because, as mentioned, I think one module per package is often too fine-grained and may be actively bad for the ecosystem. Thank you for your attention.

[c-blake]

Someone else's recent take: https://raku-advent.blog/2021/12/06/unix_philosophy_without_leftpad/ with a non-trivial (but not very conclusive) discussion at https://lobste.rs/s/nu3xyw/following_unix_philosophy_without

[ZoomRmc]

It's great that you're showing a will a to discuss and resolve the issues you yourself might not be suffering from, that's great spirit.

I personally lean towards having a lot of smaller things versus a monolith, but it's mainly a result of a specific individual traits and I have not enough experience to project the effects of these decisions on a bigger ecosystem of packages and package managers.

First of all, all things not directly related to the functioning of cligen should be moved out. If an example constitutes a whole working program with an untrivial functionality it should at least have it's own Readme. You already mentioned discoverability, and that's the main reason to separate them. Also, they can have their own issues, PRs and discussions, which is a plus. Linking to their separate repositories from cligen docs is not a problem. Also, if separate repos for each is a no-go, it's at least a nice chance for a package of programs, call them blutils, involvedpacket or industriouscoffer (or not).

On the modules which cligen currently relies on, I'm not yet acquainted enough with the internal structure to have an opinion. However, if the main concern is breaking changes, they can always be tested on a branch or "pre-release" releases. Also, ifwhen the Nim ecosystem grows, the reliance on cligen will too, and it will only make any potentially breaking decisions harder to commit to.

Another opinion I have is that smaller libs usually need to at least try to conform to the more common API in the ecosystem, while big interdependent projects will sooner or later grow their own "dialects", which make the entry bar for hacking/maintenance much higher.

[c-blake]

yourself might not be suffering from, that's great spirit.

Thanks. I try to be accommodating..sometimes I am too much so. :) All useful thoughts. We'll see what others say. I had not thought of "blutils/" or "blut/". I like it. I'll probably go with "nut/" though. If I die, someone might want to take it over. No reason `import nut/strUt` needs to ever not work even if the origin point of nut distribution changes, though I guess I might be more immortalized by "blut". Haha.

if the main concern is breaking changes, they can always be tested on a branch or "pre-release"

The concern is that moving out indirect modules will break anyone using them and for that dark code, I just cannot know how much that might be. It might be zero. I can tell just from the import structure that those modules mentioned in the discuss opener are unused *internally* and that basic CLI generation will not break and I have a test suite and all that. Anyway, the invisibility of dark code out in the wider world is the main problem with ever doing any relocation of anything (in this package or any other, really). I tried to mention this in the discuss opener. For CLI tools/command shims, dark code seems more common/likely. Simple code maybe with no deps beyond the stdlib and cligen (allowing git clone && add --path "installation") may be very common -- or it may be non-existent. But I try to keep cligen friendly towards such uses.

conform to the more common API in the ecosystem, while big interdependent

I'm not sure exactly how much of a common API or style Nim has. I usually try to be similar-ish to the way the stdlib does/names thing, but not much beyond that. None of my utility code is very fancy. Just needed for various utilities I write/wrote. Indeed, none is as fancy as the main cligen macros.

[ReneSac]

One big complaint I have, having just now read through 3+ years of changelog, is that having all those utilities here pollutes the cligen changelog. After some point I couldn't really focus on what I was reading because I didn't know if it was a new cligen functionality/change or some completely unrelated thing.

And the discoverability problem is also true. I see no link for those modules documentations.

[c-blake]

As an update/pointer to at least some doc links, I have given most of the examples/ programs a new home in bu which has like 50 programs now each with its own doc file as @ZoomRmc suggested. Utility module-wise, I made gralg for graph algos like cligen/conncomp (which was always clearly out of scope) rather than having that unified with thes. I just have to deprecate then delete/pointerize the ones here. There can/should surely be more HTML generated from nim doc than just the 3 core modules.

Re: your pollution complaint, those are fairly static areas. Aggregated dynamics are definitely one side of the problem, but there is another. For every person that finds this a problem there may be another who wants to aggregate reports (news aggregators are popular as are unified "/bin" directories..). "Software is finished" also happens much more in isolated programs than with libs. Another cost (with other benefits, too) to disaggregated dynamics is updating versions in .nimble files of consuming dependencies.

---

Not even mentioned so far is unfocused modules like, e.g. strUt.nim with A) hash functions, B) std/strutils things, C) float formatters, D) a mini-macro processor, E) tied together into uncertain number formatting. That could be 5+ modules not 1. Heck, many procs could be their own modules & packages. There is probably enough material in just my own 20 pkgs for like 500 pkgs in other viewings of the packaging problem. Then the complaint would surely become "Please aggregate!" :-) Modularity boundaries are often hard. Much is driven by expectations/opinions on "essence" which simply vary. Maybe squeaky wheels on this attend the "any" cli utility less than the gen it for me bit?

Where it comes from is cligen itself not having any non-Nim-stdlib deps. For that I have a straight line argument: "Early adoption" of new prog.langs is often simple CLI tools because of the small scope; Just git clone & nim c --path:toCligen is easier than dealing with "ecosystem stuff" like dot.nimble. QED. This mode of devel is popular in Go. Much like that mode relates to batteries included stdlibs, it also puts pressure on cligen to include batteries. Combined with dark, unpublished code using cligen/, it creates a problem. I think a better answer is a vibrant and complete Nim stdlib with trusted lieutenants for various subsystems, but we don't have that or a plan to get it.

This is not "just theory" - I have had multiple very smart & software savvy friends have to install lc with git clone because nimble failed them. Having abbrev, trie, tern be here saved them multiple steps - just to try out an ls alternative. I often feel like I have to provide alternate installation instructions.