[Discussion] haxelib replacement

[ncannasse]

At latest Haxe Summit there was several discussions regarding improving or replacing haxelib.

It seems different teams have different needs when it comes to package management, be it npm integration, private repositories, frozen dependencies & versions, etc.

We think it is not the goal of Haxe Foundation to fulfill all these needs with haxelib, as it would require a lot of work for little benefit compared to other existing solutions.

However we need to provide an easy way for developers to integrate with existing package systems (npm) or even write their own.

Unlike dynamically typed languages when only runtime matters, haxe compiler requires to be able to resolve the package/libraries class paths at compilation time, it is then necessary to have a minimal protocol that resolve these classes paths and their dependencies.

Currently the haxelib protocol is commandline based. Haxe calls haxelib path lib1 lib2 lib3 (the list of libraries that are listed with -lib <name>[@version] in compiler parameters.

haxelib then prints to its output either paths or compiler parameters (which can be extraParams.hxml).

For instance when doing haxelib path heaps I get the following answer:

D:\Projects\hxtools\heaps\
-D heaps=1.0.0
D:\Projects\hxtools\format\
-D format=3.3.0

In that case haxelib resolves also the format dependency.

This is a pretty simple protocol that could be implemented by different kind of package managers adapter. We could for instance have a haxelib_npm adapter that does something similar but looks up in a npm repository instead of a haxelib one, and would print back the same output as haxelib does.

The question is the how to have the haxe compiler to call haxelib_npm instead of the (default) haxelib

I propose that we use a dot syntax for that, so the new compiler format would be:

-lib libname[.pm][@version]

Where .pm is the package manager for this specific library.
For instance you could compile with -lib mylib.npm and the classpath would be resolved by haxelib_npm adapter, or print an error if the package adapter was not found.

Some details needs to be resolved, such as the the bootstrap code that neko uses to allow to load its ndlls by doing some runtime haxelib resolution.

[Simn]

IMO the first problem is that the compiler has to ask another tool for class paths. It would be much nicer to invert that. I haven't really thought this through, but I imagine it like this:

1. What we currently have as haxe.exe becomes haxe-compiler.exe. We can make it a big dumber because it won't have to worry about -lib at all.
2. The new haxe.exe is a small exe that figures out all the library stuff and then invokes haxe-compiler.exe with the necessary -cp and -D arguments.

[markknol]

I think Haxelib in itself isn't that bad, but just missing some important features. I have some additional thoughts.

What I think is needed in Haxelib :

1. NPM is an external service which allows to remove libraries (which happens every now and then). In general I think people don't want their Haxe libraries on NPM, but they work around the current state of haxelib. For example NPM is used to require a specific Haxe compiler version.
2. It's said before but its time to move all haxelibs out of the /lib folder inside the Haxe installation and make them user or computer globally available. I think such thing needs to go into "C:%USERS%\mark\AppData\Local\haxelibs" on Windows, on mac it is ~/Library/Application Support/haxelibs (?). This will save bandwidth+diskspace when updating/switching Haxe versions.
3. This new tiny haxe.exe is great idea. It should also be able to run a specific Haxe version. Ideally it is aware of all installed haxe versions, so it can tell/install a missing Haxe compiler version if needed.
4. Adding something like -haxe=3.4.2 as compiler argument (in hxml or cli) would set this Haxe version. And maybe -haxe:local=<path> to test nightlies.
5. I'd love to have a -lib mylib:local:<path> to point to local haxelib directory (which contains haxelib json, can have extraparams, etc). This can also be useful for tools build on top of haxelib maybe.

If this is in, then you can manage your Haxe version per project, use global and local libs. That is major benefit and allows defining a project requirements.

Less important:

• Locking versions. Ideally you can do haxelib lock build.hxml [<lib>,<lib>..] and then haxelib checks the current used versions, and replaces all -lib mylib values with current versions -lib mylib:current_version in the hxml file. I think you also want to do the same trick when haxelib lock haxelib.json to lock stuff in haxelib.json files. I now do this manually. It can be done in a separate tool, but is nicer to have build-in.
• Not very important but haxelib.json could be extended with some more info. It could describe what meta data its current library uses. For example minject uses @inject. This can be used by compiler display service so we get completion on custom meta data. \o/

These are my additions so far; I hope Haxelib will be great again!

---

I think quite some people will have interest in this topic, so I want to point out that you already can require libraries from git very easy by doing -lib mylib:git:https://github.com/user/project.git#commit, where #commit is branch, tag or commit hash.

[ibilon]

I like the idea of being able to change haxelib.

There are a libraries with a dot in their name, and I don't think we need a way to use multiple package manager in the same compilation command.

A simpler solution would be an env var, like HAXE_PM which haxe would use (or haxelib if not set) whose only requirement would be the path command.

Some details needs to be resolved, such as the the bootstrap code that neko uses to allow to load its ndlls by doing some runtime haxelib resolution.

That should have never been used, it's not like the end user of your project is going to have haxelib installed. Not sure it's worth it to add that as requirement. IMO the compiler should copy the ndlls, which is something build tools do because you need that to publish your app.

@Simn That's not mutually exclusive, the new haxe.exe could be fused with haxelib for speed, but still have a way to use something else.

I agree with @markknol, while haxelib has a lot of limitation it's not that bad and could be improved. Especially since now there's a really good update mechanism/notification (which also allow an easy use of a dev version).

People are already using alternatives so might as well make their integration with haxe easier.

[kevinresol]

IMO the first problem is that the compiler has to ask another tool for class paths. It would be much nicer to invert that. I haven't really thought this through, but I imagine it like this:

What we currently have as haxe.exe becomes haxe-compiler.exe. We can make it a big dumber because it won't have to worry about -lib at all.
The new haxe.exe is a small exe that figures out all the library stuff and then invokes haxe-compiler.exe with the necessary -cp and -D arguments.

This is exactly what haxeshim does. I think @back2dos has more info about the implementation details.

I think the general idea is that the compiler should have its own "rules" for resolving -libs into -cps without relying on any external programs. Then any package managers would only need to simply put files in the "correct place" such that the compiler could pick them up.

It would then be the manager's tasks to "lock" versions, and the compiler need not to care about the "locking mechanism" at all, as its job is merely look for files at the expected locations according to the "rules".

I think node.js uses such principle, so that we could have both npm and yarn

[ibilon]

It would then be the manager's tasks to "lock" versions, and the compiler need not to care about the "locking mechanism" at all, as its job is merely look for files at the expected locations according to the "rules".

It's already the case, the compiler has a simple rule: take all the -lib, send them to an external program and include its output.

Personally I'm not a fan of hard rules on storage, I feel it's too limiting.

But it's really two sides of the same coin, if you want to do a package manager you have to respect the protocol of the compiler. Either install location or cli.

[kevinresol]

The current rule doesn't allow custom package manager, right? We need a rule that does.

[ibilon]

Right, but could be added without too much change. As an env var, parameter or config file to store what tool to call for instance.

[kevinresol]

I am not really a big fan of having the compiler rely on another software, it simply adds risks.

[Pauan]

I'm all for improving haxelib, but I want to point out that I've been using Nix for managing my Haxe projects.

• It already supports many of the features which were listed in this thread (and many more)

• No version conflicts, multiple versions of the Haxe compiler and haxelib libraries can be used simultaneously

• It already has support for downloading libraries straight from haxelib into a globally cached directory (or using haxelib libraries locally if you wish)

• It has npm support (though I've only lightly tested it)

It's an alternative for people who want something which works now.

[ibilon]

I am not really a big fan of having the compiler rely on another software, it simply adds risks.

I don't think it's more risk than relying on something to install things correctly.
Or that we have had issues with the current setup.

[ibilon]

Couple more thoughts. Remember haxelib before newrepo, you could only use the global haxelib storage. Now imagine there was no haxelib and the compiler was simply looking for libs in the haxelib folder.

Well that means that you can't construct a package manager to do that. You simply can't since haxe will not look there. But if haxe ask the pm then you can do it, as long as you send back the correct paths.

It's easy to imagine situations where you need multiple folders like this, a global one shared between users, maybe a network disk used to share a project among collaborators, a per user folder to allow install without root access, a per project folder ...

Also they require the compiler to have more logic to deal with -lib without version, lib set as dev and can't deal with whatever new thing you want to do like for instance doing -lib foo:../libs/foo/

Hard coded rules are just that, hard coded with no evolution, it's not really known as a good practice :D Though I'll give them that they are easier to implement/use? since the compiler doesn't have to know what pm you want to use (but at another cost).

[kevinresol]

I think I like how haxeshim works.
Basically the way it resolves -lib tags is like this (i.e. the "rule"):

1. Parse arguments / hxml and look for -lib tags
2. In current directory look for a special haxe_libraries folder. (if it doesn't exists, throw an error or use some sort of global fallback)
3. For each -lib <libname>, look for haxe_libaries/libname.hxml, libname.hxml contains:
   A. -cp tags that points to its sources
   B. -lib tags for its dependencies
   C. and any extra params (e,g, -D, --macro and whatnot)
4. Recursively resolve all -lib tags by going to step 3 again.
5. The arguments should contain no -lib tags now, pass them to the "real compiler"

This is basically a dumb logic without much assumptions. It is the package managers' responsibility to write the hxml files with correct cp tags, and put the sources at the right place.

Scenarios:

• Global haxelib path: just write the global path as -cp in libname.hxml
• Network path: just write the network path as -cp in libname.hxml
• Dev path: just write the dev path as -cp in libname.hxml
• Custom path: just write the custom path as -cp in libname.hxml

And, again, writing the libname.hxml should be package manager's job.

[kevinresol]

Even if the compiler relies on some external program to resolve -lib tags, that program would still need some "data" to config themselves, that's why OP is proposing something like -lib libname@version. In that case, each pm could interpret the tag in their own ways, in other words, one package manager may not be compatible with the other one.

Think about this: If one day you find that the pm you are using is missing some features (for example it doesn't understand -lib hxnodejs@whatever-strange-version-here,maybe-just-dev://path). And you switch to another pm, then you realize it resolves the tags in completely different way. Then what?

With the npm/haxeshim approach, at least you can somehow manually put the hxmls/sources in the correct place because it is universal (rule defined by the compiler itself). But for a program, do you want to read its source code, change it, recompile it and run as your own one?

Scenario: if haxelib does not support the dev command, how could one work on libraries?

[ibilon]

And writing the libname.hxml should be package manager's job.

Oh that's better that what I imagined for hardcoded.
But that forces you to do pm install before you can compile and after every modification of a hxml right?

Not a big fan of forcing user to use a pm before they can test their small code, or adding an easily forgotten step after a modification.

one package manager may not be compatible with the other one

It's both an issue and an opportunity, to see what new feature appear.
But yeah that could be problematic, maybe.

And you switch to another pm, then you realize it resolves the tags in completely different way. Then what?

That's a valid concern, but if you change a core component of your build tool that's not entirely unexpected either.

[ncannasse]

I think there's a few conversations we should try to avoid here:

a) improving haxelib : we can add things to haxelib but as I said before it is quite hard to make sure it has all the features necessary for all haxe users

b) adopting XYZ package system : for the same reason, after having hearing about (conflicting) usages that people want to have a package system, there is a not a single-fits-all solution to be found here.

c) getting rid of the problem : haxe compiler can already be called with explicit class paths so anyone can arlready write a tool that resolves classpath and call the compiler. But this will not integrate well with IDE, compilation server, etc. we need a clear definition and integration within the compiler if we want it to work well with all tools.

d) global configuration : global configuration should be avoided if possible, as it becomes a mess to deal with libraries hosted in different package repositories, requiring to cross post and update them. My proposal allows to mix both haxelib and not-haxelib libraries

My proposal is not about deciding if you need to package your libraries this way or another. It is about allowing customization in a way that scales and integrates well.

[ibilon]

And in the end sure some stuff won't be compatible with different pm, but what can you do beside limit yourself to the current haxelib.

If your project is using haxeshim as you mentioned and contains its -scoped-hxml it's not going to work without it.
If you use a pm that allows -lib foo:1.* or -lib foo:^1.0.0 and you use haxelib/haxeshim it's not going to work.

Either every tool is exactly the same or you can't just switch without converting something. That doesn't shock me, it's like that everywhere.

If you take the nodejs example: if you use yarn and someone use npm they are not going to get the same install for the same project (since npm doesn't look into yarn.lock which yarn prioritise)

I don't see how we can have things done differently while everything working the same.

Shocker I prefer my idea :p but I think it's really a lot less intrusive while still allowing to use something else. "Just" need to change the code in haxe which does haxelib path libs to something like (HAXE_PM != null ? HAXE_PM : haxelib) path libs.

My proposal is not about deciding if you need to package your libraries this way or another. It is about allowing customization in a way that scales and integrates well.

Very close to what I'm saying so I would be fine with that, like I said though we can't use a dot since it's a valid part of the name and already used.

Not 100% sure it's worth using several pm in the same command but why not.

The issue is that it requires you to have the needed pm, though that may be already the case, and if you switch you need to change all your hxmls.

[kevinresol]

My main point is:

A program is hard to change
Just imagine if haxelib (or any future pm) has no dev command, then someone who wants to set a temporary version of a library is basically doomed. He has two choices here:

1. Modify the haxelib program by himself, that means compiling a custom version of haxelib
2. Mess with the data structure of haxelib, that means go to $HAXELIB_PATH and create a custom folder and put the temp library there.

For 1, it is just cumbersome. For 2, it's no different to manually moving stuff to where the compiler expects, as in my proposal. But the "rule" is universal as defined by the compiler.

If your project is using haxeshim as you mentioned and contains its -scoped-hxml it's not going to work without it.

haxeshim is actually an implementation of what I explained, i.e. the "rule" I have been always talking about. What it does is simply resolve -lib tags according to some simple dumb rules (already explained in previous post) into -cp tags and whatever, then send it to the real compiler. In other words, haxeshim is not the package manager. So your statement is actually not relevant.

On the other hand, lix is the pm, what it does is simply put files in the correct locations where the compiler (haxeshim) expects.

[kevinresol]

But that forces you to do pm install before you can compile and after every modification of a hxml right?

What kind of modifications you are talking about? and what hxml, the main one or the ones inside haxe_libraries folder?

If you add a new -lib in the main hxml, you will need to "install" the library though the pm, which will write a new hxml file inside haxe_libraries then download the source and put it in a correct place. Yes you need to run the pm in this case, but this is just the same as current haxelib.
Yes for haxelib if the lib already exists in global cache then you don't need to "install". But here is actually the biggest problem, the global store makes things really hard. That's why newrepo is introduced.

If you edit the hxml files inside haxe_libraries manually, then it is your responsibility to ensure it is correct. pm won't help you here.

[back2dos]

@ibilon first of all, let's get the claim out of the way that haxelib adds flexibility, because while in theory it could be true, in practice it falls flat on its face. When haxelib newrepo was added, Nicolas had to add support for it in the compiler. So much for that.

---

The way haxelib path works is way too clever, too heavy and as a result does not scale. Every time it is invoked, it must rediscover and re-resolve dependencies. For every library it must check the .current file to figure out where the relevant haxelib.json is (assuming that the indirection through .dev does not occur), then it must read and parse that JSON file to even determine the class path, then it must also check if there's an extraParams.hxml and then it must follow the dependencies, all the while detecting conflicts. This problem would get infinitely worse if we actually wanted to have support for dependency ranges, which come with a worst case O(v_1 * ... * v_n) cost, where v_k is the number of versions for lib k - so it's roughly O(averageNumberOfVesionsPerLib ^ numberOfLibs). Yes, on smaller projects you will get away with it, but as you grow it becomes unbearable.

Resolving -lib should not require nearly as much effort. To the contrary: it should be dead simple. Any package manager should make this effort once, i.e. when it actually installs the library (plus dependencies).

---

This problem is solved in haxeshim, which practically does what Simn proposes, with:

1. A few extras, like coexisting haxe versions (configured per folder) with automatic compiler server restart when switching:
   

2. A few obscure bugs in completion/diagnostics, that I don't wanna bore you with. These could easily be fixed if things were build by (or at least with the help of) people who understand these technical details.

3. An additional -scoped-hxml which is basically just historical at this point and will be removed soon(ish). A few "extensions" that haxe should offer so that IDEs can stop duplicating effort (FlashDevelop in particular).

As for flexibility, I did make a small experiment: how long to integrate npm into haxeshim? Well, it took ~30mins to get something going with <100LOC. You can see it in action here:

1. I install the library with my awesome new package manager (that NpmInstall.hx file) and then compile some random file. Works \o/
2. manually get rid of it per npm uninstall (file tree doesn't update because vscode doesn't seem to watch node_modules too eagerly ... for good reasons I suppose).
3. Naturally, compilation fails now.
4. Using the installation instructions in the hxml file (comment in the first line), haxeshim is able to download missing sources. That's what haxe --run install-libs does - it looks for all libraries where the specified class path is missing and just executes what it finds in comments that have a leading @install:
5. Compiles again \o/

Notice how in this case the particular instruction do not even require the original NpmInstall.hx itself, but just npm (which is kinda fair enough if the npm registry is your source repo and you don't want to write yet another client for it). Once NpmInstall.hx has done its work, it dumps the relevant metadata into the haxe_libraries folder (which should be versioned) including only instructions on how to download the actual sources if they are missing (this assumes of course that the metadata is constant for a specific version of a specific library - which it should be). Any "package manager" can completely get out of the way by leaving behind just something like curl https://some.registry.org/library/version/the.zip > tmp.zip && unzip tmp.zip && rm tmp.zip - this assumes of course that the URI is permanent (with "reasonable certainty" - whatever your threshold for that). Then all that's needed for installation is curl. Or you could use something similarly ubiquitous. The point is: it doesn't matter. What matters is:

There are just a few dead simple conventions to follow, and tons of flexibility fall out for free.

---

Now for a side note about lix, which is quite irrelevant to the discussion in itself, but very much so as concrete evidence that the simplistic conventions put forth by haxeshim yield highly desirable properties.

What lix does it to install libraries outside your project in a global cache, and reference them within your project-local, tracked ./haxe_libraries. The local-to-global reference is made portable by a feature of haxeshim, i.e. a user-defined environment variable that when left undefined falls back to sensible OS-specific default (Kevin is on MacOS, I'm on Windows, we can work on the same projects without any hassle). What's nice about this is that when you switch branches, there's only two options:

1. The required lib versions were already downloaded (for example because the state on the other branch was authored on the same machine), in which case you do not need to do anything to compile exactly against whatever was configured on the branch you have switched to.
2. Some source files are missing, so you get an error about that (although admittedly it could still be improved). Run haxe --run install-libs or better yet lix download (which runs the former, but also downloads the haxe compiler if the configured version is missing) and you're good. Things compile again, with exactly the stuff they were supposed to compile with. Switch back to the previous branch and you can build that state with all of its particular dependencies right away (assuming that state did compile on the current machine).

My conclusion after intensively using it for the past 8 months is that the fundamental design decision taken in the underlying infrastructure (i.e. haxeshim) "scales and integrates well" in practice.

[ibilon]

In other words, haxeshim is not the package manager.

My bad then, not too familiar with the project.

Yes you need to run the pm in this case, but this is just the same as current haxelib.

That answer the question I was about to write :D and yes I meant the main hxml.

And no you don't currently need to run haxelib if the lib is already installed.
If I do a new project with, well anything, format or hxnodejs or whatever I already have I never call haxelib.

Just imagine if haxelib (or any future pm) has no dev command, then someone who wants to set a temporary version of a library is basically doomed.

It's valid for any feature actually, what if someone wants to use any version that match 1.2.* automatically.

Either the tool you use adds it, you change tool or you have to change the hxml manually to do what you want.

--

@ibilon first of all, let's get the claim out of the way that haxelib adds flexibility

I don't think I said that, and I don't think that.

I said not having rules about where things should be could be more flexible, maybe it is maybe it's not. And what @kevinresol explained by what he means by that makes me think it might be enough flexibility indeed.

--

But like @ncannasse said let's center a bit, what we want is to use something else than haxelib because reasons, different but all valid.

Your tool is fine for you and that's all that matter.

Basically all that matter is having the possibility of using something else.
Which could be anything that has a path command that works as the compiler expect.

The way haxelib path works is way too clever

If you think that's true then use a tool that doesn't, it's fine.
It's not really the issue here.
Let's say you do HAXE_PM=lix haxe -lib foo then lix path can do what it wants and just read the haxe_libraries folder in a really quick manner, as long as it return the "expended" argument to complete the haxe command.

And IMO that's enough to start using alternative to haxelib in an easy manner, without needing to patch haxe (or include a shim) for that.

[Simn]

This rarely happens, but I'm reading what @back2dos says and basically think "yeah that's how it should work".

[Justinfront]

I am worried I spotted NPM in @back2dos post, that is not going to be setup to work on every new system that haxe should be compiled for - it's going to potentially add limitations beyond Ocaml, and it's depenancy structures fail sometimes, had problems with NPM updating etc.. in past, sorry I don't feel that Haxe needs to limited to javascript ecosystem tools, I know Luxe and Kha use npm but hoping Kha moves away from npm it's just another thing I have to install that might break that is beyond haxe control. Sure most of us end up using npm for one reason or another but I don't like it as a fundimental dependancy of Haxe if that's what is suggested?

[ibilon]

Sure most of us end up using npm for one reason or another but I don't like it as a fundimental dependancy of Haxe if that's what is suggested?

Not at all what @back2dos suggested, no need to worry

[Justinfront]

Currently I install my own projects via the haxelib git command, could haxelib be more just a link between Haxe and Git/Hg. One aspect that is sometimes problematic is that haxelib's are currently in a directory that you may only have access to as a superuser. This can be problematic as a user may want to modify them directly but not be able to access the folder or change it. If you go into a company as a freelancer frontend you may have limited admin rights so you can get someone to install stuff you require, but you can't have perm superuser access. Need to think about access rights of Haxe in general especially in relation to haxelib, as we don't want corporations banning it Haxe installation because it by passes admin too much? Totally agree that libraries in general should not be written into the same admin areas as Haxe itself, for me now it makes no odds but in a corporate enviroment it can be tricky.

[Justinfront]

I know you can change directories but I am talking about the default is not ideal for reasons explained.

[ibilon]

@Justinfront still not the issue here ;) haxelib isn't perfect and simply can't cover all use cases. Which is why we are debating on how to allow people to use something else if they want.

If you have issues on haxelib report them at https://github.com/HaxeFoundation/haxelib

Out of topic but to give you an answer you can do haxelib setup C:\Users\me\haxelib or anything, no need to put it in a admin required directory. (edit) yes the default is not optimal, raise an issue at haxelib's about that if you want

[ousado]

The option to use something else than haxelib for resolving the -lib passed to the compiler itself is a good idea, regardless of whether using (a) different tool(s) that simply provide(s) haxe with correct classpaths is the better approach, so a +1 for an HAXE_PM environment variable/commandline option.

Other than that, ultimately we should have the equivalent to rusts cargo and rustup (it sounds like haxeshim, lix and switchx cover the essential parts of those, but I haven't found the documentation to verify that impression in a quick search).

What I like a lot about the cargo approach is that a few well-chosen conventions make it very easy to get started with a project, to make sense of the structure of existing projects and to build, test and run them, all in one place. That's very, very helpful, pleasant, and would most certainly be very much appreciated by newcomers.

[ibilon]

@kevinresol @back2dos I'm curious, how does the lix/haxe_modules approach deal with multiple "main hxml" with conflicting requirement. Let's say a client.hxml with -lib foo:2.0.0 and a server.hxml still with -lib foo:1.0.0. Since it seems that the inner hxmls are named after the lib name.

[ibilon]

Some people might be perfectly fine with haxelib, at least a couple I'm sure :p

With the HAXE_PM approach you can inject lix style pm in the haxe compiler (needs a dummy path command that simply print the precalculated hxml from haxe_modules). And it'll work without any differences.
And if you like haxelib style pm then nothing changes.

If we do the haxe_modules changes, obviously lix works with that without changes.
But haxelib needs to change, not only the code but the way to use it too. So most travis scripts and mostly importantly you need to somehow inform all users.

And you can't do it without hxml? (eg haxe -lib foo) Not sure how to do an install step without one. (I could be wrong, not familiar about it)

Don't think there's a way to tell the user he forgot to do the install command to update haxe_modules, since nothing is recomputed at the compilation step.
That's going to make for some incomprehensible situations and bug report "I added -lib foo to my hxml but it's not available in my code".

Again not saying the approach is bad, just that it's a huge change, and could easily be opt in without compromise for the lix users (besides defining an env var, but that's easier than replacing the haxe exe), at least at the beginning. If everybody ends up using it we can revisit this discussion for haxe 5 ;)

[back2dos]

@ibilon You have three options:

1. lix install haxelib:foo#1.0.0 && lix install haxelib:foo#2.0.0 as foo2 and then -lib foo2 in client.hxml and -lib foo in server.hxml (choose names as you see fit). You can even compile with -lib foo -lib foo2 if it makes you happy for some reason (i.e. 2.0.0 added something cool but removed something important from 1.0.0). Will anyone ever actually need this flexibility? I have no idea. But it was very easy to add.
2. create two subdirectories, let's call them client and server each respectively having it's own haxe_modules and a build.hxml that has -cp ../src -lib foo. Then build with e.g. haxe --cwd client build.hxml (in haxeshim it would actually currently be haxe -scoped-hxml client/build.hxml, but that's a technicality). That way you can even have different haxe versions for server and client. Will anyone need that? No idea either.
3. make it so that both builds can use the same version of foo, as this will probably be the better choice in the long run anyway.

Pick your poison.

[ibilon]

The first is very heavy for the users, but the second is actually not that bad. Still a bit manual but I guess that's the limit of the approach.
Ofc 3 is better ;) my imaginary use case was in progress upgrade.

[EricBishton]

So From an IDE perspective, things should look like this:

   IDE -> FrontEndTool -> HaxeCompiler

FrontEndTool can be haxelib, lix, (did I hear languageServer?) or whatever. The IDE should be configurable to know which FrontEndTool to run, and any such tool should have a common interface to do a few things:

• Install/Remove/Manage libraries (however and wherever it wants.)
• Run a compile (resolving all libraries to classpaths), passing all library arguments (defines, etc), returning all messages back on its standard output channels -- stderr, stdout).
• Give a dry run output (for debugging purposes)
  And the complex ones:
• Pass-through of compiler services, such as completion, find, etc. This definitely needs a protocol -- and it's own design document.

The HaxeCompiler should NOT call back out to the library tools. -lib should be removed as an input option to the compiler. Most other options should only be allowed through a protocol/configuration file -- like an hxml file but using a better format. (Indeed, IMO, the compiler input should become convenient for machine readability and should not allow a normal command line -- that's the front-end tool's job. Only the name of an input file should be needed for compiler debug testing, but the compiler could take its input from stdin.)

The design should allow for this eventual configuration, where the IDE replaces the FrontEndTool:

   IDE -> HaxeCompiler

All of that said, it is true that libraries still need a common protocol to describe them so there can be some interoperability between the tools. Standard things like defines, external libraries, and dependencies (and versions) should be codified; of course, using an extensible structure like XML or JSON.

[ibilon]

The HaxeCompiler should NOT call back out to the library tools. -lib should be removed as an input option to the compiler.

Sure in theory that's better, but pragmatically is it worth changing everybody's workflow?

Either you need to tell people to use another tool instead of calling haxe directly, or you make a small haxe.exe that does that.
Knowing that right now that's just 45 lines in the compiler https://github.com/HaxeFoundation/haxe/blob/development/src/compiler/main.ml#L110-L155

I don't think the package manager should double as a build tool.

Most other options should only be allowed through a protocol/configuration file -- like an hxml file but using a better format.

Is that worth it? You'll prevent using haxe directly, which may not be the most used way, but still useful to have. I don't see what the benefit of that approach is, if you could expend on that @EricBishton

[EricBishton]

Sure in theory that's better, but pragmatically is it worth changing everybody's workflow?

I wouldn't change the workflow. Haxe.exe should become a "compiler driver" (as you say).

I don't think the package manager should double as a build tool.

Well, a package manager (any package manager) by definition is a build tool by virtue of its being necessary to create and manage an environment that other build tools rely on. Whether it's involved in any particular compile run is neither here nor there.

I don't think the package manager should double as a build tool.

Fair enough. They can certainly be separate and I have no issue with that. But, then the compiler has to become aware of libraries and their configuration, which is what haxelib is trying to isolate it from. The problem is that calling back to haxelib is bad practice, even if it follows the design principle of separation of concerns (which is good).

There are many issues with haxelib in the field. In particular IDEs have an issue because the current tool depends upon environment variables which are generally not set at the time the IDE is launched (generally from an icon). For example, haxelib returns a huge negative number and displays nothing when either its own directory or neko is missing from the path. I perennially get issues from users who can't get things to work when the root issue is their environment. Perhaps the IDEs can do better by noting all required variables and/or providing them when missing. That causes other issues around noting when they change and how to make the user revisit their configuration to resolve the problems.

I truly don't mind preventing the user from running the back-end compiler directly. (User's likely don't know that every Microsoft compiler actually works this way -- a front end that deals with options and even 'pre-compiles' to a psuedo-code, which then forks the real compiler, much like Haxe's back-end compilers today.) They can deal with a compiler-driver and it can be called Haxe. The difference is that the compiler can be driven from any type of program/library, and I believe that Haxe language-server should become one.

In the end, the IDEs (certainly IDEA) wants to become the compiler driver, for several of reasons: performance during compiles; better ability to watch/control compiler output (put up progress bars, better error detection, ability to halt a compile and whatnot); ability to issue commands (such as completion) serially more quickly; and (the holy grail) incremental compilation, so that compile errors can be detected, marked, and erased as quickly as the user types. In essence, treating the compiler as a part of itself, rather than an external tool chain.

[Simn]

Closed and locked in favor of #30.