Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
[Discussion] haxelib replacement #29
At latest Haxe Summit there was several discussions regarding improving or replacing haxelib.
It seems different teams have different needs when it comes to package management, be it npm integration, private repositories, frozen dependencies & versions, etc.
We think it is not the goal of Haxe Foundation to fulfill all these needs with haxelib, as it would require a lot of work for little benefit compared to other existing solutions.
However we need to provide an easy way for developers to integrate with existing package systems (npm) or even write their own.
Unlike dynamically typed languages when only runtime matters, haxe compiler requires to be able to resolve the package/libraries class paths at compilation time, it is then necessary to have a minimal protocol that resolve these classes paths and their dependencies.
Currently the haxelib protocol is commandline based. Haxe calls
haxelib then prints to its output either paths or compiler parameters (which can be extraParams.hxml).
For instance when doing
In that case haxelib resolves also the
This is a pretty simple protocol that could be implemented by different kind of package managers adapter. We could for instance have a
The question is the how to have the haxe compiler to call
I propose that we use a dot syntax for that, so the new compiler format would be:
Some details needs to be resolved, such as the the bootstrap code that neko uses to allow to load its ndlls by doing some runtime haxelib resolution.
IMO the first problem is that the compiler has to ask another tool for class paths. It would be much nicer to invert that. I haven't really thought this through, but I imagine it like this:
I think Haxelib in itself isn't that bad, but just missing some important features. I have some additional thoughts.
What I think is needed in Haxelib :
If this is in, then you can manage your Haxe version per project, use global and local libs. That is major benefit and allows defining a project requirements.
These are my additions so far; I hope Haxelib will be great again!
I think quite some people will have interest in this topic, so I want to point out that you already can require libraries from git very easy by doing
I like the idea of being able to change haxelib.
There are a libraries with a dot in their name, and I don't think we need a way to use multiple package manager in the same compilation command.
A simpler solution would be an env var, like
That should have never been used, it's not like the end user of your project is going to have haxelib installed. Not sure it's worth it to add that as requirement. IMO the compiler should copy the ndlls, which is something build tools do because you need that to publish your app.
@Simn That's not mutually exclusive, the new haxe.exe could be fused with haxelib for speed, but still have a way to use something else.
I agree with @markknol, while haxelib has a lot of limitation it's not that bad and could be improved. Especially since now there's a really good update mechanism/notification (which also allow an easy use of a dev version).
People are already using alternatives so might as well make their integration with haxe easier.
I think the general idea is that the compiler should have its own "rules" for resolving
It would then be the manager's tasks to "lock" versions, and the compiler need not to care about the "locking mechanism" at all, as its job is merely look for files at the expected locations according to the "rules".
I think node.js uses such principle, so that we could have both
It's already the case, the compiler has a simple rule: take all the -lib, send them to an external program and include its output.
Personally I'm not a fan of hard rules on storage, I feel it's too limiting.
But it's really two sides of the same coin, if you want to do a package manager you have to respect the protocol of the compiler. Either install location or cli.
I'm all for improving haxelib, but I want to point out that I've been using Nix for managing my Haxe projects.
It's an alternative for people who want something which works now.
Couple more thoughts. Remember haxelib before newrepo, you could only use the global haxelib storage. Now imagine there was no haxelib and the compiler was simply looking for libs in the haxelib folder.
Well that means that you can't construct a package manager to do that. You simply can't since haxe will not look there. But if haxe ask the pm then you can do it, as long as you send back the correct paths.
It's easy to imagine situations where you need multiple folders like this, a global one shared between users, maybe a network disk used to share a project among collaborators, a per user folder to allow install without root access, a per project folder ...
Also they require the compiler to have more logic to deal with
Hard coded rules are just that, hard coded with no evolution, it's not really known as a good practice :D Though I'll give them that they are easier to implement/use? since the compiler doesn't have to know what pm you want to use (but at another cost).
I think I like how haxeshim works.
This is basically a dumb logic without much assumptions. It is the package managers' responsibility to write the hxml files with correct cp tags, and put the sources at the right place.
And, again, writing the libname.hxml should be package manager's job.
Even if the compiler relies on some external program to resolve
Think about this: If one day you find that the pm you are using is missing some features (for example it doesn't understand
With the npm/haxeshim approach, at least you can somehow manually put the hxmls/sources in the correct place because it is universal (rule defined by the compiler itself). But for a program, do you want to read its source code, change it, recompile it and run as your own one?
Scenario: if haxelib does not support the
Oh that's better that what I imagined for hardcoded.
Not a big fan of forcing user to use a pm before they can test their small code, or adding an easily forgotten step after a modification.
It's both an issue and an opportunity, to see what new feature appear.
That's a valid concern, but if you change a core component of your build tool that's not entirely unexpected either.
I think there's a few conversations we should try to avoid here:
a) improving haxelib : we can add things to haxelib but as I said before it is quite hard to make sure it has all the features necessary for all haxe users
b) adopting XYZ package system : for the same reason, after having hearing about (conflicting) usages that people want to have a package system, there is a not a single-fits-all solution to be found here.
c) getting rid of the problem : haxe compiler can already be called with explicit class paths so anyone can arlready write a tool that resolves classpath and call the compiler. But this will not integrate well with IDE, compilation server, etc. we need a clear definition and integration within the compiler if we want it to work well with all tools.
d) global configuration : global configuration should be avoided if possible, as it becomes a mess to deal with libraries hosted in different package repositories, requiring to cross post and update them. My proposal allows to mix both haxelib and not-haxelib libraries
My proposal is not about deciding if you need to package your libraries this way or another. It is about allowing customization in a way that scales and integrates well.
And in the end sure some stuff won't be compatible with different pm, but what can you do beside limit yourself to the current haxelib.
If your project is using haxeshim as you mentioned and contains its
Either every tool is exactly the same or you can't just switch without converting something. That doesn't shock me, it's like that everywhere.
If you take the nodejs example: if you use yarn and someone use npm they are not going to get the same install for the same project (since npm doesn't look into yarn.lock which yarn prioritise)
I don't see how we can have things done differently while everything working the same.
Shocker I prefer my idea :p but I think it's really a lot less intrusive while still allowing to use something else. "Just" need to change the code in haxe which does
Very close to what I'm saying so I would be fine with that, like I said though we can't use a dot since it's a valid part of the name and already used.
Not 100% sure it's worth using several pm in the same command but why not.
The issue is that it requires you to have the needed pm, though that may be already the case, and if you switch you need to change all your hxmls.
My main point is:
A program is hard to change
For 1, it is just cumbersome. For 2, it's no different to manually moving stuff to where the compiler expects, as in my proposal. But the "rule" is universal as defined by the compiler.
haxeshim is actually an implementation of what I explained, i.e. the "rule" I have been always talking about. What it does is simply resolve
On the other hand, lix is the pm, what it does is simply put files in the correct locations where the compiler (haxeshim) expects.
What kind of modifications you are talking about? and what hxml, the main one or the ones inside
If you add a new
If you edit the hxml files inside
@ibilon first of all, let's get the claim out of the way that haxelib adds flexibility, because while in theory it could be true, in practice it falls flat on its face. When
This problem is solved in haxeshim, which practically does what Simn proposes, with:
As for flexibility, I did make a small experiment: how long to integrate npm into haxeshim? Well, it took ~30mins to get something going with <100LOC. You can see it in action here:
Notice how in this case the particular instruction do not even require the original
There are just a few dead simple conventions to follow, and tons of flexibility fall out for free.
Now for a side note about lix, which is quite irrelevant to the discussion in itself, but very much so as concrete evidence that the simplistic conventions put forth by haxeshim yield highly desirable properties.
What lix does it to install libraries outside your project in a global cache, and reference them within your project-local, tracked
My conclusion after intensively using it for the past 8 months is that the fundamental design decision taken in the underlying infrastructure (i.e. haxeshim) "scales and integrates well" in practice.
My bad then, not too familiar with the project.
That answer the question I was about to write :D and yes I meant the main hxml.
And no you don't currently need to run haxelib if the lib is already installed.
It's valid for any feature actually, what if someone wants to use any version that match
Either the tool you use adds it, you change tool or you have to change the hxml manually to do what you want.
I don't think I said that, and I don't think that.
I said not having rules about where things should be could be more flexible, maybe it is maybe it's not. And what @kevinresol explained by what he means by that makes me think it might be enough flexibility indeed.
But like @ncannasse said let's center a bit, what we want is to use something else than haxelib because reasons, different but all valid.
Your tool is fine for you and that's all that matter.
Basically all that matter is having the possibility of using something else.
If you think that's true then use a tool that doesn't, it's fine.
And IMO that's enough to start using alternative to haxelib in an easy manner, without needing to patch haxe (or include a shim) for that.
Currently I install my own projects via the haxelib git command, could haxelib be more just a link between Haxe and Git/Hg. One aspect that is sometimes problematic is that haxelib's are currently in a directory that you may only have access to as a superuser. This can be problematic as a user may want to modify them directly but not be able to access the folder or change it. If you go into a company as a freelancer frontend you may have limited admin rights so you can get someone to install stuff you require, but you can't have perm superuser access. Need to think about access rights of Haxe in general especially in relation to haxelib, as we don't want corporations banning it Haxe installation because it by passes admin too much? Totally agree that libraries in general should not be written into the same admin areas as Haxe itself, for me now it makes no odds but in a corporate enviroment it can be tricky.
@Justinfront still not the issue here ;) haxelib isn't perfect and simply can't cover all use cases. Which is why we are debating on how to allow people to use something else if they want.
If you have issues on haxelib report them at https://github.com/HaxeFoundation/haxelib
Out of topic but to give you an answer you can do
The option to use something else than haxelib for resolving the -lib passed to the compiler itself is a good idea, regardless of whether using (a) different tool(s) that simply provide(s) haxe with correct classpaths is the better approach, so a +1 for an HAXE_PM environment variable/commandline option.
Other than that, ultimately we should have the equivalent to rusts cargo and rustup (it sounds like haxeshim, lix and switchx cover the essential parts of those, but I haven't found the documentation to verify that impression in a quick search).
What I like a lot about the cargo approach is that a few well-chosen conventions make it very easy to get started with a project, to make sense of the structure of existing projects and to build, test and run them, all in one place. That's very, very helpful, pleasant, and would most certainly be very much appreciated by newcomers.
Some people might be perfectly fine with haxelib, at least a couple I'm sure :p
With the HAXE_PM approach you can inject lix style pm in the haxe compiler (needs a dummy path command that simply print the precalculated hxml from haxe_modules). And it'll work without any differences.
If we do the haxe_modules changes, obviously lix works with that without changes.
And you can't do it without hxml? (eg
Don't think there's a way to tell the user he forgot to do the install command to update haxe_modules, since nothing is recomputed at the compilation step.
Again not saying the approach is bad, just that it's a huge change, and could easily be opt in without compromise for the lix users (besides defining an env var, but that's easier than replacing the haxe exe), at least at the beginning. If everybody ends up using it we can revisit this discussion for haxe 5 ;)
@ibilon You have three options:
Pick your poison.
So From an IDE perspective, things should look like this:
FrontEndTool can be haxelib, lix, (did I hear languageServer?) or whatever. The IDE should be configurable to know which FrontEndTool to run, and any such tool should have a common interface to do a few things:
The HaxeCompiler should NOT call back out to the library tools. -lib should be removed as an input option to the compiler. Most other options should only be allowed through a protocol/configuration file -- like an hxml file but using a better format. (Indeed, IMO, the compiler input should become convenient for machine readability and should not allow a normal command line -- that's the front-end tool's job. Only the name of an input file should be needed for compiler debug testing, but the compiler could take its input from stdin.)
The design should allow for this eventual configuration, where the IDE replaces the FrontEndTool:
All of that said, it is true that libraries still need a common protocol to describe them so there can be some interoperability between the tools. Standard things like defines, external libraries, and dependencies (and versions) should be codified; of course, using an extensible structure like XML or JSON.
Sure in theory that's better, but pragmatically is it worth changing everybody's workflow?
Either you need to tell people to use another tool instead of calling haxe directly, or you make a small haxe.exe that does that.
I don't think the package manager should double as a build tool.
Is that worth it? You'll prevent using haxe directly, which may not be the most used way, but still useful to have. I don't see what the benefit of that approach is, if you could expend on that @EricBishton
I wouldn't change the workflow. Haxe.exe should become a "compiler driver" (as you say).
Well, a package manager (any package manager) by definition is a build tool by virtue of its being necessary to create and manage an environment that other build tools rely on. Whether it's involved in any particular compile run is neither here nor there.
Fair enough. They can certainly be separate and I have no issue with that. But, then the compiler has to become aware of libraries and their configuration, which is what haxelib is trying to isolate it from. The problem is that calling back to haxelib is bad practice, even if it follows the design principle of separation of concerns (which is good).
There are many issues with haxelib in the field. In particular IDEs have an issue because the current tool depends upon environment variables which are generally not set at the time the IDE is launched (generally from an icon). For example, haxelib returns a huge negative number and displays nothing when either its own directory or neko is missing from the path. I perennially get issues from users who can't get things to work when the root issue is their environment. Perhaps the IDEs can do better by noting all required variables and/or providing them when missing. That causes other issues around noting when they change and how to make the user revisit their configuration to resolve the problems.
I truly don't mind preventing the user from running the back-end compiler directly. (User's likely don't know that every Microsoft compiler actually works this way -- a front end that deals with options and even 'pre-compiles' to a psuedo-code, which then forks the real compiler, much like Haxe's back-end compilers today.) They can deal with a compiler-driver and it can be called Haxe. The difference is that the compiler can be driven from any type of program/library, and I believe that Haxe language-server should become one.
In the end, the IDEs (certainly IDEA) wants to become the compiler driver, for several of reasons: performance during compiles; better ability to watch/control compiler output (put up progress bars, better error detection, ability to halt a compile and whatnot); ability to issue commands (such as completion) serially more quickly; and (the holy grail) incremental compilation, so that compile errors can be detected, marked, and erased as quickly as the user types. In essence, treating the compiler as a part of itself, rather than an external tool chain.