New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[stage1] Haxelib replacement #30

Closed
Simn opened this Issue Sep 30, 2017 · 41 comments

Comments

Projects
None yet
@Simn
Member

Simn commented Sep 30, 2017

Foreword

Issues in this repositories tend to get bogged down in long-winded discussions without much of a result. We would like to try a slightly different approach which is outlined here: https://github.com/HaxeFoundation/haxe-evolution/wiki/Haxe-Projects

We are in Stage 1 here, which means that comments should be short and to the point as mentioned in the document. If you want to discuss the procedure, please open a separate issue.

Now on to the actual project.


Haxelib replacement

Project lead: Simn
Due date for stage 1: 2017-10-14 (2 weeks)

We are looking to replace haxelib with a better package manager solution.

Please share your ideas and concerns in a concise manner. Please also +1/-1 individual comments that you agree/disagree with.

@Simn Simn self-assigned this Sep 30, 2017

@Simn

This comment has been minimized.

Member

Simn commented Sep 30, 2017

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

@kevinresol

This comment has been minimized.

kevinresol commented Sep 30, 2017

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

That means the compiler should be able to transform -lib tags into non-lib tags on its own, with the resolving logic being simple yet generic.

@elsassph

This comment has been minimized.

elsassph commented Sep 30, 2017

Criteria:

  • semver dependency management, and solution to handle conflicts as we can have only one version of a lib,
  • global cache and local libs similarly to npm/yarn,
  • support for git / file dependencies.
@damoebius

This comment has been minimized.

damoebius commented Sep 30, 2017

NPM isn't enough ?

@ibilon

This comment has been minimized.

Member

ibilon commented Sep 30, 2017

The user should be able to use an alternative package manager (without making change to or replacing the haxe exe).

@ousado

This comment has been minimized.

ousado commented Sep 30, 2017

The compiler should expose an interface for frontends like CLIs, IDEs, vscode-style language servers, and so on.

@ousado

This comment has been minimized.

ousado commented Sep 30, 2017

Whatever ends up as standard mechanism to invoke the compiler for one-shot compilations must be very fast, there should be no noticeable difference to invoking the compiler directly (in other words start-up times like that of nodejs or even the JVM are out of the question)

@ousado

This comment has been minimized.

ousado commented Sep 30, 2017

The standard mechanism to invoke the compiler will be a first point of contact for newcomers to the language. As such it should provide an outstanding user experience and documentation can't be an afterthought.

@ibilon

This comment has been minimized.

Member

ibilon commented Sep 30, 2017

We shouldn't change the current workflow and so not introduce any steps between changing a hxml and running it/doing a haxe command by hand.

@ibilon

This comment has been minimized.

Member

ibilon commented Sep 30, 2017

A migration plan is required:

  • what happens to the already installed libraries? do they need to be reinstalled?
  • do we keep the haxelib online repository as is? if not what about usernames?
@ousado

This comment has been minimized.

ousado commented Sep 30, 2017

Beyond building projects the standard mechanism to invoke the compiler should provide a standardized way to run other common tasks like tests, benchmarks, updating dependencies, etc. (much in the spirit of cargo).

@ibilon

This comment has been minimized.

Member

ibilon commented Sep 30, 2017

haxelib.json should be extended, proposed new fields (non exhaustive):

  • repository field (to go with the current website field)
  • listing the metas/defines of the library
  • haxe version compatibility
@kevinresol

This comment has been minimized.

kevinresol commented Sep 30, 2017

That means the compiler should be able to transform -lib tags into non-lib tags on its own, with the resolving logic being simple yet generic.

Because -lib is the only thing that cannot be "consumed directly" by the compiler.

@MangelMaxime

This comment has been minimized.

MangelMaxime commented Sep 30, 2017

  • Support lock files so we have reproductive build system
@ibilon

This comment has been minimized.

Member

ibilon commented Sep 30, 2017

The following fields should be removed from haxelib.json (or its replacements) and dealt on the library repository website:

  • description
  • website
  • contributors
  • tags

since it doesn't make sense to update those by making new releases.

@ousado

This comment has been minimized.

ousado commented Sep 30, 2017

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

@elsassph

This comment has been minimized.

elsassph commented Sep 30, 2017

I know @damoebius' suggestion to use npm has been firmly downvoted but it isn't that silly - alternatively it would be interesting to have a npm-compatible haxelib API so we could be really interesting to have.

@EricBishton

This comment has been minimized.

Member

EricBishton commented Sep 30, 2017

Configuration must be thoroughly documented and transparent. On-disk storage such as hidden .dev, and .current files should be discouraged and instead replaced by a documented configuration file.

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

@Simn

This comment has been minimized.

Member

Simn commented Sep 30, 2017

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

Out of scope (except updates, but that's obvious). This is about a package manager, not a complete build tool, which would be one level higher.

@ousado

This comment has been minimized.

ousado commented Sep 30, 2017

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

That's too restrictive. While some package managers might agree on one database layout, others might not. A package manager using e.g. Nix as a backend will require a different layout than one based on a git repo layout, both of which would be reasonable choices to make.

If what you mean is a common layout for a staging area, as well as for ex- and importing a package database to allow migration between package managers, OTOH, that'd be a good idea.

@ousado

This comment has been minimized.

ousado commented Sep 30, 2017

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

Out of scope (except updates, but that's obvious). This is about a package manager, not a complete build tool, which would be one level higher.

I was under the impression we're talking about all the (future) components required for the workflows that haxelib is currently involved with, not just its role as a package manager.

If this is about package management only, I think we should look at the more principled approaches - so basically read A modular package manager architecture, conclude that we're probably not going to come up with something better, and hence "do whatever opam does".

@EricBishton

This comment has been minimized.

Member

EricBishton commented Sep 30, 2017

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

That's too restrictive. While some package managers might agree on one database layout, others might not. A package manager using e.g. Nix as a backend will require a different layout than one based on a git repo layout, both of which would be reasonable choices to make.

It is and it isn't. While I respect the decision of writers of the extended or alternative package managers to create the solutions they think most appropriate, I believe that the canonical package manager (the one delivered as part of the Haxe toolset) must set an open standard that other tools can inter-operate with.

@HaxeFoundation HaxeFoundation deleted a comment from sangohan Oct 1, 2017

@HaxeFoundation HaxeFoundation deleted a comment from jcward Oct 1, 2017

@HaxeFoundation HaxeFoundation deleted a comment from jcward Oct 1, 2017

@Simn

This comment has been minimized.

Member

Simn commented Oct 1, 2017

@sangohan @jcward: I removed your comments because they are way too wordy/not actionable. I've pasted them here so they are not lost: https://gist.github.com/Simn/2b13809d830cfd16dd5ba2f9c97cac6b

@Matan

This comment has been minimized.

Matan commented Oct 1, 2017

  • Maintain a local cache of downloaded libs.
  • Symlink cached libs to projects to save space. (flag to control behaviour and/or force local install)
  • Support "globally" installing libs.
  • Support licenses sub method to output all dependency licenses much like PHP's composer licenses.
@kiroukou

This comment has been minimized.

kiroukou commented Oct 1, 2017

I recently discovered https://lib.haxe.org/p/hmm library which does a pretty nice job on that part with current limitations.

For me, the current haxelib version isn't really far from what I need, but here is my wishlist :

  • Having a way to abstract the lib source (git, svn, local, ...)
  • Having 2 scopes of lib loading : global and local
  • Recursive loading (a lib depending itself on an other, should be able to load it automatically). That leads to a version conflict resolver !
  • Having/Keeping a comprehensive synthax
@jasononeil

This comment has been minimized.

jasononeil commented Oct 1, 2017

It should be possible to open a directory that a project is in, and when I type haxe build.hxml it automatically uses the configured version of Haxe for the project, and the right set of haxelibs/dependencies.

(This one might be controversial but I thought I'd throw it in!)

@jasononeil

This comment has been minimized.

jasononeil commented Oct 1, 2017

If I install a new dependency, upgrade a dependency, or change my dependencies in any way, this should show up in a way that is tracked in a file in my version control, and can be used to restore the same state on a different computer.

(Similar to a yarn.lock or haxe_libraries/*.hxml in haxeshim/lix)

@clemos

This comment has been minimized.

clemos commented Oct 1, 2017

I think we should deprecate haxelib run to avoid depending on neko.

@Simn

This comment has been minimized.

Member

Simn commented Oct 1, 2017

I think we should deprecate haxelib run to avoid depending on neko.

Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

@ibilon

This comment has been minimized.

Member

ibilon commented Oct 1, 2017

(Similar to a yarn.lock or haxe_libraries/*.hxml in haxeshim/lix)

Can't put that in version control since it contains computer specific path into it.
EDIT: not actually the case, see @jasononeil comment below

As an alternative to a new file (like yarn.lock) maybe lock the versions directly in the hxml?

@jasononeil

This comment has been minimized.

jasononeil commented Oct 1, 2017

@ibilon here is how lix deals with it, no absolute paths:

# @install: lix --silent download "haxelib:hxnodejs#4.0.9" into hxnodejs/4.0.9/haxelib
-D hxnodejs=4.0.9
-cp ${HAXESHIM_LIBCACHE}/hxnodejs/4.0.9/haxelib/src
-D nodejs
--macro allowPackage('sys')
--macro _hxnodejs.VersionWarning.include()

Here is how yarn.lock deals with it (basically, it doesn't, it just uses relative paths):

angular-cookies@^1.6.4:
  version "1.6.4"
  resolved "https://registry.yarnpkg.com/angular-cookies/-/angular-cookies-1.6.4.tgz#c28f3f6aac7a9826c1e45f1d6807240036e5b26d"
@back2dos

This comment has been minimized.

Member

back2dos commented Oct 1, 2017

The package manager should use secure protocols (HTTPS vs. HTTP) for downloading packages (I found this to place limitations on the choice of a suitable runtime).

@markknol

This comment has been minimized.

Member

markknol commented Oct 1, 2017

From Haxe code it should still be possible to know which libs are used. so doing #if (libname > 4). It would also be useful to be able to know which classes come which library (in macros/ documentation xmls/dependency dumps etc)

@jgranick

This comment has been minimized.

jgranick commented Oct 2, 2017

I have many feelings on haxelib, but quickly l want to share some "wish list" items:

  • Support for multiple remote repositories
  • Support for cascading local library paths
  • Support for wildcard versions (IE -lib lime:3.1.*)
  • Support for installing shortcut commands for tools
  • Self-updating

Multiple remote repositories would allow companies to generate their own internal versions and distribute more easily, or enable "beta" or "dev" channel repositories.

Maintaining a single class path directory would not be reasonable for dealing with source files. Similarly, it would be nice to be able to add multiple custom library paths -- similar to how class paths are resolved.

Support for wildcard versions would really, really help (especially if semver is respected). You can request library version 1.0.*, allowing for patch releases. This is a great middle-solution between "Give me version 1.0.4 exactly" or "Give me any installed version", and a frequently requested feature for OpenFL

As a cherry on top, it would be nice if libraries could install command aliases, such as lime for haxelib run lime or as3hx for haxelib run as3hx

The best package manager in the world will need updates. I would like to see a library that is self-updated when you run a command similar to haxelib update

@elsassph

This comment has been minimized.

elsassph commented Oct 2, 2017

@clemos haxelib run isn't a problem per se, but it should be extended to other runtimes, and be configurable in haxelib.json.

@elsassph

This comment has been minimized.

elsassph commented Oct 2, 2017

Caching libs is good, using symlinks is bad: someone could mistakenly modify a file and it would affect the cache / other projects.
Libs should be copied locally in the project (unless optional CI optimisation to use cache).

@clemos

This comment has been minimized.

clemos commented Oct 3, 2017

@Simn @elsassph I agree it's not so much about about deprecating haxelib run itself, but rather about deprecating run.n in favor of Run.hx ;)

@bablukid

This comment has been minimized.

bablukid commented Oct 3, 2017

The package manager should allow custom fields in haxelib.json
It would be very usefull for librairies which are tied to a specific framework or runtime.
i.e : For externs, we could provide a link to the original library and infos about how to integrate it in your haxe project
i.e : a web framework which loads plugins (haxelibs) that have various assets ( like models, controllers, templates and translation files )

@skial skial referenced this issue Oct 4, 2017

Closed

Haxe Roundup 402 #437

@andyli

This comment has been minimized.

Member

andyli commented Oct 13, 2017

I would like to point out that it is important to consider how to handle "native code" (could be compiled ndll, C/C++/JS/... compiled libraries or source files).

One reason is that it is wasteful to download binaries of all architectures. e.g. Lime's ndll folder is 108M, but if you're only using Windows, you actually only need the 5.1M ndll/Windows folder.

We should also have some guidelines regarding to how to store the info of those "native" things, e.g. where is the original source, what param is used when building (use of -PIC/-O3/debug info), any additional dependencies that is not bundled etc.

@kevinresol

This comment has been minimized.

kevinresol commented Oct 14, 2017

Build tools (hxcpp,hxjava,hxcs) should not rely on haxelib too.
They can be invoked by Context.onAfterGenerate or haxe --run

@Simn Simn closed this Oct 14, 2017

@HaxeFoundation HaxeFoundation locked and limited conversation to collaborators Oct 14, 2017

@Simn

This comment has been minimized.

Member

Simn commented Oct 14, 2017

Stage 1 has concluded, I'll prepare the discussion part for Monday.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.