Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[stage2] Haxelib replacement #34
Issues in this repositories tend to get bogged down in long-winded discussions without much of a result. We would like to try a slightly different approach which is outlined here: https://github.com/HaxeFoundation/haxe-evolution/wiki/Haxe-Projects
We are in Stage 2 here, which means that discussion should be limited to the items from stage 1. Below is the result of stage 1, ordered descending by rating (= Upvotes - Downvotes - Off topic votes). I am only listing items which had more than 5 rating. This doesn't mean that all other aspects should be completely ignored (especially late entries which might not have had enough time to get votes), but the focus should still be on the list below.
Now on to the actual project.
We are looking to replace haxelib with a better package manager solution.
General guidelines (higher level)
@back2dos: Which items of this feature set does lix not cover (at the moment)?
We will look into supporting https in the interpreter. From @ncannasse regarding that:
A problem with that is that it would require all the dependencies. One nice thing about run.n is that you don't need anything else. For instance, with Dox we provide a run.n so that users don't need all its dependencies to run it, only to compile it.
I'm currently working on a Haxe binary output which could help regarding that.
Either that's built-in in the compiler, and we better make it right the first time considering the compiler's release cycle (even if that's better now).
Or that falls to the user call a separate tool to obtain the information required for compilation.
Not too familiar with the concept of lock files, do they allow that dependency change tracking?
@Simn it would probably be easier if the items from stage 1 had a number id.
I had the same concern, but this "interpreting an .hx file" idea is interesting. It could alleviate these concerns.
Now this I don't understand. Of course you want HTTPS on the package authoring / upload side, but on the distribution / download side? HTTPS inhibits possible caching layers (reverse proxies, intermediate caches) and causes extra dependencies for the package manager. This is a public package repo, right? No login / secret info required. So what's the motivation for HTTPS?
MITM attacks for one. However unlikely for anyone to spend their time on it, you just shouldn't transfer any code that will be run on your pc over http. It's just too easy to interfere with plain http data on public wifi or similar situations. Https should not be optional. I also don't see any reason for it to inhibit caching layers.
Many of the features are actually covered just by haxeshim's stupid simple conventions, that could be trivially baked into the compiler (and should be IMO). From top to bottom, theses are the points that are yet open:
To start, I think haxelib is a simple tool, it's easy to understand, and it does one thing (somewhat) well. Once you understand it and its limitations, you'll create tooling around it and live with it.
The idea of
I'm less clear on lix, node, and npm joining the party. Seems like an awful lot of (JS-leaning) tooling prerequisites. Now I have to understand how NPM works and package.json and whatnot. It's a lot to take in, gotta look at it more.
Just to play devil's advocate, what could I do today with a simple script, checked into my repo, that gives me a reproducible setup with haxelib as it stands:
# Use a project-local haxelib haxelib setup ./haxelib # or ./haxe_libraries # [copy or symlink or unzip or wget, my choice] some cached libs to ./haxelib # Clone some git haxelibs at exact commits: haxelib git hxcpp-debugger https://github.com/HaxeFoundation/hxcpp-debugger
All this thinking about the very basics of Haxe lib management makes me wonder -- would it make haxelib better to simply remove features? What if...
As for "version of Haxe" -- I consider this slightly out of scope. I use very simple, conventional, (and sadly OS-specific) symlinks to select versions of Haxe and it works like a charm.
FWIW understanding NPM is not such a bad choice ;)
What I have maintained for the past three years is that the conventions implemented in haxeshim should be built into the compiler. Beyond that point, I don't think we have to agree on anything. That's the beauty of it. People may or may not choose to use lix. If someone comes forward with a better package manager, I'm more than happy to use it, because I don't particularly enjoy working on such a thing. It's just that I have a desperate need for such a tool (and I believe that our community does share that need, even though they might not necessarily realize it ... more on that later).
(only read this if you're interested in lix)To understand lix, it's important to note that its first and foremost goal is reliability. The second goal is - yet again - reliability. And the third goal - believe it or not - is reliability once more. With those three goals in mind, after spending a few weekends exploring the many options that Haxe presents, I settled nodejs as a runtime because:
The first point is something the compiler team could fix. As for the rest, I guess lix could also be written against
Now, with node as an interim (?) runtime, the obvious choice for installation is
That said: beyond using npm to install lix, you don't need npm. You don't need a package.json. It provides an additional layer of scoping and reliability, so I guess it's advisable to use (at least until such time that lix is properly tested and thus sufficiently guarded against regressions). I also used that setup for the workshop, because I wanted a self-contained setup that installs all tools locally (including haxeshim, which otherwise would override your
I'm sorry about the somewhat Spartan nature of lix. It is a tool solely driven by the need of those who use it. Fortunately, Jason set out to improve the documentation.
Uhm, nope. This is not reproducible in the slightest. Let's set aside the fact that if your machine has no git it fails and if it has cygwin git, it fails - better yet - for totally obscure reasons. The main point is: if you run that script a couple of months apart, you get two different versions of the code, where the old one might only be Haxe 3 compatible and the new one might only be Haxe 4 compatible. And that's just with one library. If you have 100 of them, the chance of winding up with compatible versions after running the script is practically 0. So you will have to write all the particular tags/commits into your setup script by hand and maintain it by hand and remember to rerun it, whenever you switch branches.
Sure, one can live with it. I documented a pretty equivalent approach years ago. Unfortunately, it just turns out that if something is optional and not automated, it doesn't get done (at least not sufficiently well).
That is why lix fully automates this part, for all dependencies, including the transitive ones.
I am generally a big fan of removing features. That said, if the alternative is to rely on git only, my advise would be to use git submodules as Rob does. If I din't always forget to update submodules when switching, I guess that's what I'd go with myself. But it might work perfectly for you.
As mentioned though, git is not universally supported. What's more: it's not a reliable source of 3rd party code, because the owner may alter/delete the history or the whole repo. In the coming months, lix will gain a create-only (no deletion, no alteration except deprecation) registry intended as the primary source of packages, which will prevent things like the npm fiasco to unfold in its core ecosystem.
Sure. It works for you, on your machine and I'm quite optimistic that on the whole many people have found some way to arrange themselves with haxe and haxelib. That's great, but it still leaves room for us to progress as a collective.
The main reason why I do see library and language version management as an indivisible unit is usage.
First, there's the matter of having a replicable setup, that will work the same on your CI and on every team member's machine (regardless of OS, regardless of which branch they are on, etc.). Big teams care about this. It's fair to say that the reason this debate was tabled is that at the foundation meeting on the summit the foundation partners agreed on such a need. And it involves the compiler version too. If a patch is landed for something we need, we want to be able to use it across all machines without much hassle - and only on a specific branch to see what other problems might occur. With lix's support for nightlies, we can.
Second, and I think this is just as important (if not more), I see much value in a solid basis for creating tutorials that don't just completely break within months. There's always talk about how Haxe needs more tutorials and what not, but very little thought is given to the technical prerequisites to make them useful, such as making sure that the code actually works for the user. I believe there's a huge difference between a code example that is slightly outdated but compiles just fine, because setting it up was straight forward, and a code example that is slightly outdated and greets you with 500 different errors, half of which come deep out of some macro, that was broken between haxe 3.2.1 (that the tutorial was written for) and 3.4.4 (that the user has on their machine). It might even be detrimental to lock versions in such a case, because the library that the example is based on most likely had a haxe 3.4.4 compatibility update in the mean time that just didn't get installed due to version locking.
So, if you look at these two version management problems not from a purely technical perspective, but through the prism of meaningful use cases, they superpose almost wholly. Given how often concerns are voiced about reducing friction for newcomers, this could make an important difference. My advise is not to waste such an opportunity.
Adding just the library convention into the compiler with fallback to haxelib borders on trivial. It's easy to do and breaks nothing, providing a clear migration path. Because of that, it's probably even worth adding to Haxe 3.
As for Haxe 4, I think the idea of supporting parallel Haxe versions at least deserves thorough consideration. If it's too hard to add, that's one thing. But it shouldn't be dismissed merely on the grounds that "it's out of scope". It isn't.
Thanks for the thoughtful responses, @back2dos.
Oh, I meant to specify a specific commit URL.
Yes, I think I was just realizing that. So haxeshim (hence, Haxe itself) would expect those conventional directory structures, and you're free to populate them however you choose. I like this a lot (this was the core of my "simple script" idea -- just tell Haxe to look in some conventional folder, and beyond that, I'll script a reproducible setup.)
There does need to be some name / version conventions on the lib side, though, doesn't there? I suppose a library will choose to support lix, thus using lix conventions? It might be nice to be as generic as possible in that specification, so other tools can parse versions and dependencies.
Alright, well anyway, just trying to follow along. I appreciate the dialog. Cheers!
Is this still open? The due date was 8 days ago. I think I'll stick my $0.02 in anyway.
It's important to note that having reproducible builds is important to me, so I'm going to advocate anything that I think would bring haxe closer to having reproducible builds. I also have no clue how the internals of the haxe compiler or haxelib work, so bear with me.