New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[stage2] Haxelib replacement #34

Closed
Simn opened this Issue Oct 16, 2017 · 19 comments

Comments

Projects
None yet
@Simn
Copy link
Member

Simn commented Oct 16, 2017

Foreword

Issues in this repositories tend to get bogged down in long-winded discussions without much of a result. We would like to try a slightly different approach which is outlined here: https://github.com/HaxeFoundation/haxe-evolution/wiki/Haxe-Projects

We are in Stage 2 here, which means that discussion should be limited to the items from stage 1. Below is the result of stage 1, ordered descending by rating (= Upvotes - Downvotes - Off topic votes). I am only listing items which had more than 5 rating. This doesn't mean that all other aspects should be completely ignored (especially late entries which might not have had enough time to get votes), but the focus should still be on the list below.

Now on to the actual project.


Haxelib replacement

Project lead: Simn
Stage 1 issue: #30
Due date for stage 2: 2017-11-6 (3 weeks)

We are looking to replace haxelib with a better package manager solution.

Feature set

  • (17)
    • semver dependency management, and solution to handle conflicts as we can have only one version of a lib,
    • global cache and local libs similarly to npm/yarn,
    • support for git / file dependencies.
  • (16) The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.
  • (16) The package manager should use secure protocols (HTTPS vs. HTTP) for downloading packages (I found this to place limitations on the choice of a suitable runtime).
  • (15) Support lock files so we have reproductive build system
  • (15) If I install a new dependency, upgrade a dependency, or change my dependencies in any way, this should show up in a way that is tracked in a file in my version control, and can be used to restore the same state on a different computer.
  • (12) Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.
  • (12) haxelib.json should be extended, proposed new fields (non exhaustive):
    • repository field (to go with the current website field)
    • listing the metas/defines of the library
    • haxe version compatibility
  • (11) It should be possible to open a directory that a project is in, and when I type haxe build.hxml it automatically uses the configured version of Haxe for the project, and the right set of haxelibs/dependencies.
  • (10) From Haxe code it should still be possible to know which libs are used. so doing #if (libname > 4). It would also be useful to be able to know which classes come which library (in macros/ documentation xmls/dependency dumps etc)
  • (8) The compiler should expose an interface for frontends like CLIs, IDEs, vscode-style language servers, and so on.

General guidelines (higher level)

  • (15) The user should be able to use an alternative package manager (without making change to or replacing the haxe exe).
  • (13) Configuration must be thoroughly documented and transparent. On-disk storage such as hidden .dev, and .current files should be discouraged and instead replaced by a documented configuration file.
    The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.
  • (10) The standard mechanism to invoke the compiler will be a first point of contact for newcomers to the language. As such it should provide an outstanding user experience and documentation can't be an afterthought.
  • (8) A migration plan is required:
    • what happens to the already installed libraries? do they need to be reinstalled?
    • do we keep the haxelib online repository as is? if not what about usernames?
@Simn

This comment has been minimized.

Copy link
Member

Simn commented Oct 16, 2017

@back2dos: Which items of this feature set does lix not cover (at the moment)?

The package manager should use secure protocols (HTTPS vs. HTTP) for downloading packages (I found this to place limitations on the choice of a suitable runtime).

We will look into supporting https in the interpreter. From @ncannasse regarding that:

<nicolas> now that we have properly working ssl support for HL/Neko we should use these for OCaml/Interp as well
<nicolas> should be straightforward to backport HL ssl.hdll to ocaml FFI

Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

A problem with that is that it would require all the dependencies. One nice thing about run.n is that you don't need anything else. For instance, with Dox we provide a run.n so that users don't need all its dependencies to run it, only to compile it.

I'm currently working on a Haxe binary output which could help regarding that.

@ibilon

This comment has been minimized.

Copy link
Member

ibilon commented Oct 16, 2017

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

Either that's built-in in the compiler, and we better make it right the first time considering the compiler's release cycle (even if that's better now).

Or that falls to the user call a separate tool to obtain the information required for compilation.
Hardly an improvement in my opinion, how many time will one think "why doesn't it work, everything is good, oh I forgot to call the tool to tell the compiler I've changed my hxml file"?

Support lock files so we have reproductive build system

If I install a new dependency, upgrade a dependency, or change my dependencies in any way, this should show up in a way that is tracked in a file in my version control, and can be used to restore the same state on a different computer.

Not too familiar with the concept of lock files, do they allow that dependency change tracking?

--

@Simn it would probably be easier if the items from stage 1 had a number id.

@jcward

This comment has been minimized.

Copy link

jcward commented Oct 16, 2017

Either that's built-in in the compiler (ocaml? slow release cycle?) ... Or that falls to the user call a separate tool ... Hardly an improvement in my opinion.

I had the same concern, but this "interpreting an .hx file" idea is interesting. It could alleviate these concerns.

HTTPS

Now this I don't understand. Of course you want HTTPS on the package authoring / upload side, but on the distribution / download side? HTTPS inhibits possible caching layers (reverse proxies, intermediate caches) and causes extra dependencies for the package manager. This is a public package repo, right? No login / secret info required. So what's the motivation for HTTPS?

@benmerckx

This comment has been minimized.

Copy link

benmerckx commented Oct 16, 2017

So what's the motivation for HTTPS?

MITM attacks for one. However unlikely for anyone to spend their time on it, you just shouldn't transfer any code that will be run on your pc over http. It's just too easy to interfere with plain http data on public wifi or similar situations. Https should not be optional. I also don't see any reason for it to inhibit caching layers.

@skial skial referenced this issue Oct 18, 2017

Closed

Haxe Roundup 404 #441

@back2dos

This comment has been minimized.

Copy link
Member

back2dos commented Oct 19, 2017

Which items of this feature set does lix not cover (at the moment)?

Many of the features are actually covered just by haxeshim's stupid simple conventions, that could be trivially baked into the compiler (and should be IMO). From top to bottom, theses are the points that are yet open:

  • semver dependency management, and solution to handle conflicts as we can have only one version of a lib,

    This is largely the matter of having a registry that actually exposes this information.

  • support for git / file dependencies.

    Partial. See lix-pm/lix.client#19, lix-pm/lix.client#10 and lix-pm/lix.client#5 ... I haven't done much here, because lix places reproducible setups above all. Those features run counter my personal mission. If lix is considered as the "official" haxelib replacement, I am willing to compromise of course.

    One option is also to use mixed resolution strategy, which runs haxelib path for any libraries that cannot be resolved per haxeshim convention, in which case haxelib dev will work just fine.

  • Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

    No script running. Because lix currently relies on npm and that already has it's own way of doing things, the necessity did not arise to rush towards something half baked. This is an interesting subject that should be explored in depth.

  • haxelib.json should be extended, proposed new fields (non exhaustive):

    • repository field (to go with the current website field)
    • listing the metas/defines of the library
    • haxe version compatibility

    A non-exhaustive makes for a very vague feature specification :P

    Lix does not alter the haxelib.json in any way. That said, I quite like the idea of listing metas/defines. The question is whether or not we wish to process such information in the compiler or whether it's purely for documentation purposes (in which case nothing needs to be done in the package manager).

  • The standard mechanism to invoke the compiler will be a first point of contact for newcomers to the language. As such it should provide an outstanding user experience and documentation can't be an afterthought.

    If "outstanding" is the benchmark here, then there's certainly more work to be done. The haxe command works as before, but the error messages could certainly be improved.

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Oct 27, 2017

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

This should apply on build tools as well (e.g. hxcpp, hxjava, hxcs)

@Simn

This comment has been minimized.

Copy link
Member

Simn commented Oct 30, 2017

I have extended stage 2 time by a week because I won't be available until then anyway.

@jcward

This comment has been minimized.

Copy link

jcward commented Nov 7, 2017

I've watched the London meetup, and tried the MVCoconut example, and I'm trying to digest the concepts.

To start, I think haxelib is a simple tool, it's easy to understand, and it does one thing (somewhat) well. Once you understand it and its limitations, you'll create tooling around it and live with it.

The idea of haxeshim seems fine enough -- a convention with a scoped directory structure containing library references, and a global cache of (referenced) versioned libraries. With a place for neko and haxe versions to boot. To me, the best things this provides beyond haxelib is: my dependencies are defined (and committed) with each project.

I'm less clear on lix, node, and npm joining the party. Seems like an awful lot of (JS-leaning) tooling prerequisites. Now I have to understand how NPM works and package.json and whatnot. It's a lot to take in, gotta look at it more.

Just to play devil's advocate, what could I do today with a simple script, checked into my repo, that gives me a reproducible setup with haxelib as it stands:

init_local_haxelib.sh

# Use a project-local haxelib
haxelib setup ./haxelib # or ./haxe_libraries
# [copy or symlink or unzip or wget, my choice] some cached libs to ./haxelib
# Clone some git haxelibs at exact commits:
haxelib git hxcpp-debugger https://github.com/HaxeFoundation/hxcpp-debugger
@jcward

This comment has been minimized.

Copy link

jcward commented Nov 7, 2017

All this thinking about the very basics of Haxe lib management makes me wonder -- would it make haxelib better to simply remove features? What if...

  • It only supported only git, and only a project-local ./haxe_libraries repository.
  • The library's git repos contain haxelib.json and optionally versions.json specifying commits in history that are different versions of the lib.
  • A project would define its dependencies as either
    1. git repo URL (aka, latest, whatever version is in haxelib.json), or
    2. git repo URL + major version (auto updates minors), or
    3. git rev-specific URL (obviously, this is a specific version)
  • instead of haxelib path running every compile, path is always ./haxe_libraries/libName
  • haxe --init-libs is only run once (to resolve, download, and optionally check for minor updates.) This interprets <haxe builtin>/NewHaxelib.hx, which sets up ./haxe_libraries, or you can override with ./NewHaxelib.hx if you like
  • Global lib cache is enabled via an environment variable -- e.g. $GLOBAL_DIR/libName/version (or ~/.haxelib, I smell haxelib backward compatibility here...)

As for "version of Haxe" -- I consider this slightly out of scope. I use very simple, conventional, (and sadly OS-specific) symlinks to select versions of Haxe and it works like a charm.

@elnabo

This comment has been minimized.

Copy link

elnabo commented Nov 7, 2017

  • Supporting git only would be a pain to use some libraries that requires compilation like hxcpp
@back2dos

This comment has been minimized.

Copy link
Member

back2dos commented Nov 7, 2017

I'm less clear on lix, node, and npm joining the party. Seems like an awful lot of (JS-leaning) tooling prerequisites. Now I have to understand how NPM works and package.json and whatnot. It's a lot to take in, gotta look at it more.

FWIW understanding NPM is not such a bad choice ;)

What I have maintained for the past three years is that the conventions implemented in haxeshim should be built into the compiler. Beyond that point, I don't think we have to agree on anything. That's the beauty of it. People may or may not choose to use lix. If someone comes forward with a better package manager, I'm more than happy to use it, because I don't particularly enjoy working on such a thing. It's just that I have a desperate need for such a tool (and I believe that our community does share that need, even though they might not necessarily realize it ... more on that later).

(only read this if you're interested in lix) To understand lix, it's important to note that its first and foremost goal is reliability. The second goal is - yet again - reliability. And the third goal - believe it or not - is reliability once more. With those three goals in mind, after spending a few weekends exploring the many options that Haxe presents, I settled nodejs as a runtime because:
  • HTTPS: many companies and individuals increasingly value security

  • compared to the nodejs APIs, haxe's sys package is dismal. It doesn't even provide symlinking, permissions, proper control over child processes (in particular their environment and their I/O). Not a showstopper, but definitely a PITA.

  • nodejs is a mature piece of technology, with wide adoption and serious backing. It is battle tested and is relatively sure to still exist 5 years from now. I guess the next best thing we have is neko, which is being deprecated. There's other sys backends, but without any tests there's no way to know how well they will work. Writing a package manager (and thus wrestling with three different OSes) is hard enough without having to worry about the APIs that you're using.

  • (bonus) everything is streaming. This perceivably improves user experience. To quote @thomasuster on the matter:

    So I noticed this actually
    Like the libs were installing crazy fast
    But I wasn't sure how
    then I was scared it was like cacheing my private repos on some server

    (hint: there's no caching going on)

The first point is something the compiler team could fix. As for the rest, I guess lix could also be written against sys with a few concessions and a lot of debugging (the best solution by the way would be the libuv bindings for eval I keep pestering @Simn about). IMO for the time being the pragmatic thing to do is to use nodejs. There's more than enough design decisions to figure out and that's my primary focus.

Now, with node as an interim (?) runtime, the obvious choice for installation is npm install lix.pm -g. This choice immediately proved useful, because the handful of people who started using lix are scattered over all three major OSes, which makes npm the most common package manager (of course nothing stands in the way of adding a package for brew or chocolatey or whatever).

That said: beyond using npm to install lix, you don't need npm. You don't need a package.json. It provides an additional layer of scoping and reliability, so I guess it's advisable to use (at least until such time that lix is properly tested and thus sufficiently guarded against regressions). I also used that setup for the workshop, because I wanted a self-contained setup that installs all tools locally (including haxeshim, which otherwise would override your haxe command). Still, you can limit any contact you have with npm just to installation of haxeshim (which hopefully will be assimilated by the compiler), switchx (which is likely to merge with lix) and lix itself. If we make progress here, installing lix is the one thing you'll have to do to use lix, which I suppose is fair enough.

I'm sorry about the somewhat Spartan nature of lix. It is a tool solely driven by the need of those who use it. Fortunately, Jason set out to improve the documentation.

Just to play devil's advocate, what could I do today with a simple script, checked into my repo, that gives me a reproducible setup with haxelib as it stands:

[...]

Uhm, nope. This is not reproducible in the slightest. Let's set aside the fact that if your machine has no git it fails and if it has cygwin git, it fails - better yet - for totally obscure reasons. The main point is: if you run that script a couple of months apart, you get two different versions of the code, where the old one might only be Haxe 3 compatible and the new one might only be Haxe 4 compatible. And that's just with one library. If you have 100 of them, the chance of winding up with compatible versions after running the script is practically 0. So you will have to write all the particular tags/commits into your setup script by hand and maintain it by hand and remember to rerun it, whenever you switch branches.

Sure, one can live with it. I documented a pretty equivalent approach years ago. Unfortunately, it just turns out that if something is optional and not automated, it doesn't get done (at least not sufficiently well).

That is why lix fully automates this part, for all dependencies, including the transitive ones.

All this thinking about the very basics of Haxe lib management makes me wonder -- would it make haxelib better to simply remove features?

I am generally a big fan of removing features. That said, if the alternative is to rely on git only, my advise would be to use git submodules as Rob does. If I din't always forget to update submodules when switching, I guess that's what I'd go with myself. But it might work perfectly for you.

As mentioned though, git is not universally supported. What's more: it's not a reliable source of 3rd party code, because the owner may alter/delete the history or the whole repo. In the coming months, lix will gain a create-only (no deletion, no alteration except deprecation) registry intended as the primary source of packages, which will prevent things like the npm fiasco to unfold in its core ecosystem.

As for "version of Haxe" -- I consider this slightly out of scope. I use very simple, conventional, (and sadly OS-specific) symlinks and it works like a charm.

Sure. It works for you, on your machine and I'm quite optimistic that on the whole many people have found some way to arrange themselves with haxe and haxelib. That's great, but it still leaves room for us to progress as a collective.

The main reason why I do see library and language version management as an indivisible unit is usage.

First, there's the matter of having a replicable setup, that will work the same on your CI and on every team member's machine (regardless of OS, regardless of which branch they are on, etc.). Big teams care about this. It's fair to say that the reason this debate was tabled is that at the foundation meeting on the summit the foundation partners agreed on such a need. And it involves the compiler version too. If a patch is landed for something we need, we want to be able to use it across all machines without much hassle - and only on a specific branch to see what other problems might occur. With lix's support for nightlies, we can.

Second, and I think this is just as important (if not more), I see much value in a solid basis for creating tutorials that don't just completely break within months. There's always talk about how Haxe needs more tutorials and what not, but very little thought is given to the technical prerequisites to make them useful, such as making sure that the code actually works for the user. I believe there's a huge difference between a code example that is slightly outdated but compiles just fine, because setting it up was straight forward, and a code example that is slightly outdated and greets you with 500 different errors, half of which come deep out of some macro, that was broken between haxe 3.2.1 (that the tutorial was written for) and 3.4.4 (that the user has on their machine). It might even be detrimental to lock versions in such a case, because the library that the example is based on most likely had a haxe 3.4.4 compatibility update in the mean time that just didn't get installed due to version locking.

So, if you look at these two version management problems not from a purely technical perspective, but through the prism of meaningful use cases, they superpose almost wholly. Given how often concerns are voiced about reducing friction for newcomers, this could make an important difference. My advise is not to waste such an opportunity.


Adding just the library convention into the compiler with fallback to haxelib borders on trivial. It's easy to do and breaks nothing, providing a clear migration path. Because of that, it's probably even worth adding to Haxe 3.

As for Haxe 4, I think the idea of supporting parallel Haxe versions at least deserves thorough consideration. If it's too hard to add, that's one thing. But it shouldn't be dismissed merely on the grounds that "it's out of scope". It isn't.

@jcward

This comment has been minimized.

Copy link

jcward commented Nov 7, 2017

Thanks for the thoughtful responses, @back2dos.

This is not reproducible in the slightest. ... if you run that script a couple of months apart, you get two different versions of the code.

Oh, I meant to specify a specific commit URL.

the conventions implemented in haxeshim should be built into the compiler. Beyond that point, I don't think we have to agree on anything. That's the beauty of it.

Yes, I think I was just realizing that. So haxeshim (hence, Haxe itself) would expect those conventional directory structures, and you're free to populate them however you choose. I like this a lot (this was the core of my "simple script" idea -- just tell Haxe to look in some conventional folder, and beyond that, I'll script a reproducible setup.)

There does need to be some name / version conventions on the lib side, though, doesn't there? I suppose a library will choose to support lix, thus using lix conventions? It might be nice to be as generic as possible in that specification, so other tools can parse versions and dependencies.

Alright, well anyway, just trying to follow along. I appreciate the dialog. Cheers! 🍺

@desttinghim

This comment has been minimized.

Copy link

desttinghim commented Nov 14, 2017

Is this still open? The due date was 8 days ago. I think I'll stick my $0.02 in anyway.

It's important to note that having reproducible builds is important to me, so I'm going to advocate anything that I think would bring haxe closer to having reproducible builds. I also have no clue how the internals of the haxe compiler or haxelib work, so bear with me.

  • Haxeshim should definitely be the default behavior of the haxe compiler. This reduces the coupling between the two, which is important for allowing haxe to integrate into other package managers such as npm, nix, and guix in a cleaner way.
  • Switchx (compiler version switching/locking) is also a good thing. Without a mechanism like this, build files and/or README's must include the version of the compiler and steps to install it.
  • Lock files are a definite yes, for convenience more than anything. These should include the compiler versions. The implementation for this seems like it would be a little hairy, but it is also an essential part of making builds reproducible. A possible implementation could be having the haxe command as a script that interprets the version that needs to be (either defined in the environment, in a lock file, or passed in explicitly) that then hands off control to a specific version (e.g. haxe3-4-1).

EDIT:

  • I don't like the idea of relying on lix as the standard package manager. It's nice enough, but I would prefer something that doesn't rely on the NPM/Node ecosystem. Preferably it could be compiled to C++, but that's just a personal preference.
@aliokan

This comment has been minimized.

Copy link

aliokan commented Feb 26, 2018

Who can push this subject on the next step? @ncannasse

The package manager is one of the basics in the tool chain.
We regularly reach the limit with haxelib on projects with a lot of dependencies.

I really hope it will be part of the next major release. :)

@jonasmalacofilho

This comment has been minimized.

Copy link

jonasmalacofilho commented Feb 26, 2018

I would also like to bring back a point previously made by @andyli:

it is important to consider how to handle "native code" (could be compiled ndll, C/C++/JS/... compiled libraries or source files).

@ncannasse

This comment has been minimized.

Copy link
Member

ncannasse commented Feb 26, 2018

The topic has been quite discussed, thank you to all the participants.
We are currently recruiting for HF (see https://haxe.org/blog/hf-is-recruiting/) and new haxelib will be one of the tasks to prioritize in our upcoming developments.

@Simn

This comment has been minimized.

Copy link
Member

Simn commented Apr 17, 2018

Sorry about that, that was bad timing on my part. I'll close this issue for now.

... to be continued

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Jul 19, 2018

Just to make sure this is not forgotten

@kevinresol

This comment has been minimized.

Copy link

kevinresol commented Sep 14, 2018

Is this happening in haxe 4?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment