Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[stage1] Haxelib replacement #30

Closed
Simn opened this issue Sep 30, 2017 · 41 comments
Closed

[stage1] Haxelib replacement #30

Simn opened this issue Sep 30, 2017 · 41 comments
Assignees

Comments

@Simn
Copy link
Member

@Simn Simn commented Sep 30, 2017

Foreword

Issues in this repositories tend to get bogged down in long-winded discussions without much of a result. We would like to try a slightly different approach which is outlined here: https://github.com/HaxeFoundation/haxe-evolution/wiki/Haxe-Projects

We are in Stage 1 here, which means that comments should be short and to the point as mentioned in the document. If you want to discuss the procedure, please open a separate issue.

Now on to the actual project.


Haxelib replacement

Project lead: Simn
Due date for stage 1: 2017-10-14 (2 weeks)

We are looking to replace haxelib with a better package manager solution.

Please share your ideas and concerns in a concise manner. Please also +1/-1 individual comments that you agree/disagree with.

@Simn Simn self-assigned this Sep 30, 2017
@Simn
Copy link
Member Author

@Simn Simn commented Sep 30, 2017

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

@kevinresol
Copy link

@kevinresol kevinresol commented Sep 30, 2017

The Haxe compiler should not have to call a separate tool to obtain the information required for compilation.

That means the compiler should be able to transform -lib tags into non-lib tags on its own, with the resolving logic being simple yet generic.

@elsassph
Copy link

@elsassph elsassph commented Sep 30, 2017

Criteria:

  • semver dependency management, and solution to handle conflicts as we can have only one version of a lib,
  • global cache and local libs similarly to npm/yarn,
  • support for git / file dependencies.
@damoebius
Copy link

@damoebius damoebius commented Sep 30, 2017

NPM isn't enough ?

@ibilon
Copy link
Member

@ibilon ibilon commented Sep 30, 2017

The user should be able to use an alternative package manager (without making change to or replacing the haxe exe).

@ousado
Copy link

@ousado ousado commented Sep 30, 2017

The compiler should expose an interface for frontends like CLIs, IDEs, vscode-style language servers, and so on.

@ousado
Copy link

@ousado ousado commented Sep 30, 2017

Whatever ends up as standard mechanism to invoke the compiler for one-shot compilations must be very fast, there should be no noticeable difference to invoking the compiler directly (in other words start-up times like that of nodejs or even the JVM are out of the question)

@ousado
Copy link

@ousado ousado commented Sep 30, 2017

The standard mechanism to invoke the compiler will be a first point of contact for newcomers to the language. As such it should provide an outstanding user experience and documentation can't be an afterthought.

@ibilon
Copy link
Member

@ibilon ibilon commented Sep 30, 2017

We shouldn't change the current workflow and so not introduce any steps between changing a hxml and running it/doing a haxe command by hand.

@ibilon
Copy link
Member

@ibilon ibilon commented Sep 30, 2017

A migration plan is required:

  • what happens to the already installed libraries? do they need to be reinstalled?
  • do we keep the haxelib online repository as is? if not what about usernames?
@ousado
Copy link

@ousado ousado commented Sep 30, 2017

Beyond building projects the standard mechanism to invoke the compiler should provide a standardized way to run other common tasks like tests, benchmarks, updating dependencies, etc. (much in the spirit of cargo).

@ibilon
Copy link
Member

@ibilon ibilon commented Sep 30, 2017

haxelib.json should be extended, proposed new fields (non exhaustive):

  • repository field (to go with the current website field)
  • listing the metas/defines of the library
  • haxe version compatibility
@kevinresol
Copy link

@kevinresol kevinresol commented Sep 30, 2017

That means the compiler should be able to transform -lib tags into non-lib tags on its own, with the resolving logic being simple yet generic.

Because -lib is the only thing that cannot be "consumed directly" by the compiler.

@MangelMaxime
Copy link

@MangelMaxime MangelMaxime commented Sep 30, 2017

  • Support lock files so we have reproductive build system
@ibilon
Copy link
Member

@ibilon ibilon commented Sep 30, 2017

The following fields should be removed from haxelib.json (or its replacements) and dealt on the library repository website:

  • description
  • website
  • contributors
  • tags

since it doesn't make sense to update those by making new releases.

@ousado
Copy link

@ousado ousado commented Sep 30, 2017

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

@elsassph
Copy link

@elsassph elsassph commented Sep 30, 2017

I know @damoebius' suggestion to use npm has been firmly downvoted but it isn't that silly - alternatively it would be interesting to have a npm-compatible haxelib API so we could be really interesting to have.

@EricBishton
Copy link
Member

@EricBishton EricBishton commented Sep 30, 2017

Configuration must be thoroughly documented and transparent. On-disk storage such as hidden .dev, and .current files should be discouraged and instead replaced by a documented configuration file.

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

@Simn
Copy link
Member Author

@Simn Simn commented Sep 30, 2017

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

Out of scope (except updates, but that's obvious). This is about a package manager, not a complete build tool, which would be one level higher.

@ousado
Copy link

@ousado ousado commented Sep 30, 2017

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

That's too restrictive. While some package managers might agree on one database layout, others might not. A package manager using e.g. Nix as a backend will require a different layout than one based on a git repo layout, both of which would be reasonable choices to make.

If what you mean is a common layout for a staging area, as well as for ex- and importing a package database to allow migration between package managers, OTOH, that'd be a good idea.

@ousado
Copy link

@ousado ousado commented Sep 30, 2017

Interesting, what are the arguments against tool test, tool bench, tool update .. ?

Out of scope (except updates, but that's obvious). This is about a package manager, not a complete build tool, which would be one level higher.

I was under the impression we're talking about all the (future) components required for the workflows that haxelib is currently involved with, not just its role as a package manager.

If this is about package management only, I think we should look at the more principled approaches - so basically read A modular package manager architecture, conclude that we're probably not going to come up with something better, and hence "do whatever opam does".

@EricBishton
Copy link
Member

@EricBishton EricBishton commented Sep 30, 2017

The final goal is to make it so that alternative library managers can use/manipulate the same library database without having to know the details of the previous manager.

That's too restrictive. While some package managers might agree on one database layout, others might not. A package manager using e.g. Nix as a backend will require a different layout than one based on a git repo layout, both of which would be reasonable choices to make.

It is and it isn't. While I respect the decision of writers of the extended or alternative package managers to create the solutions they think most appropriate, I believe that the canonical package manager (the one delivered as part of the Haxe toolset) must set an open standard that other tools can inter-operate with.

@HaxeFoundation HaxeFoundation deleted a comment from sangohan Oct 1, 2017
@HaxeFoundation HaxeFoundation deleted a comment from jcward Oct 1, 2017
@HaxeFoundation HaxeFoundation deleted a comment from jcward Oct 1, 2017
@Simn
Copy link
Member Author

@Simn Simn commented Oct 1, 2017

@sangohan @jcward: I removed your comments because they are way too wordy/not actionable. I've pasted them here so they are not lost: https://gist.github.com/Simn/2b13809d830cfd16dd5ba2f9c97cac6b

@Matan
Copy link

@Matan Matan commented Oct 1, 2017

  • Maintain a local cache of downloaded libs.
  • Symlink cached libs to projects to save space. (flag to control behaviour and/or force local install)
  • Support "globally" installing libs.
  • Support licenses sub method to output all dependency licenses much like PHP's composer licenses.
@kiroukou
Copy link

@kiroukou kiroukou commented Oct 1, 2017

I recently discovered https://lib.haxe.org/p/hmm library which does a pretty nice job on that part with current limitations.

For me, the current haxelib version isn't really far from what I need, but here is my wishlist :

  • Having a way to abstract the lib source (git, svn, local, ...)
  • Having 2 scopes of lib loading : global and local
  • Recursive loading (a lib depending itself on an other, should be able to load it automatically). That leads to a version conflict resolver !
  • Having/Keeping a comprehensive synthax
@jasononeil
Copy link

@jasononeil jasononeil commented Oct 1, 2017

It should be possible to open a directory that a project is in, and when I type haxe build.hxml it automatically uses the configured version of Haxe for the project, and the right set of haxelibs/dependencies.

(This one might be controversial but I thought I'd throw it in!)

@jasononeil
Copy link

@jasononeil jasononeil commented Oct 1, 2017

If I install a new dependency, upgrade a dependency, or change my dependencies in any way, this should show up in a way that is tracked in a file in my version control, and can be used to restore the same state on a different computer.

(Similar to a yarn.lock or haxe_libraries/*.hxml in haxeshim/lix)

@clemos
Copy link

@clemos clemos commented Oct 1, 2017

I think we should deprecate haxelib run to avoid depending on neko.

@Simn
Copy link
Member Author

@Simn Simn commented Oct 1, 2017

I think we should deprecate haxelib run to avoid depending on neko.

Don't have to deprecate it, but it shouldn't require a run.n. It could, by convention, make haxe interpret a Run.hx script instead.

@ibilon
Copy link
Member

@ibilon ibilon commented Oct 1, 2017

(Similar to a yarn.lock or haxe_libraries/*.hxml in haxeshim/lix)

Can't put that in version control since it contains computer specific path into it.
EDIT: not actually the case, see @jasononeil comment below

As an alternative to a new file (like yarn.lock) maybe lock the versions directly in the hxml?

@jasononeil
Copy link

@jasononeil jasononeil commented Oct 1, 2017

@ibilon here is how lix deals with it, no absolute paths:

# @install: lix --silent download "haxelib:hxnodejs#4.0.9" into hxnodejs/4.0.9/haxelib
-D hxnodejs=4.0.9
-cp ${HAXESHIM_LIBCACHE}/hxnodejs/4.0.9/haxelib/src
-D nodejs
--macro allowPackage('sys')
--macro _hxnodejs.VersionWarning.include()

Here is how yarn.lock deals with it (basically, it doesn't, it just uses relative paths):

angular-cookies@^1.6.4:
  version "1.6.4"
  resolved "https://registry.yarnpkg.com/angular-cookies/-/angular-cookies-1.6.4.tgz#c28f3f6aac7a9826c1e45f1d6807240036e5b26d"
@back2dos
Copy link
Member

@back2dos back2dos commented Oct 1, 2017

The package manager should use secure protocols (HTTPS vs. HTTP) for downloading packages (I found this to place limitations on the choice of a suitable runtime).

@markknol
Copy link
Member

@markknol markknol commented Oct 1, 2017

From Haxe code it should still be possible to know which libs are used. so doing #if (libname > 4). It would also be useful to be able to know which classes come which library (in macros/ documentation xmls/dependency dumps etc)

@jgranick
Copy link

@jgranick jgranick commented Oct 2, 2017

I have many feelings on haxelib, but quickly l want to share some "wish list" items:

  • Support for multiple remote repositories
  • Support for cascading local library paths
  • Support for wildcard versions (IE -lib lime:3.1.*)
  • Support for installing shortcut commands for tools
  • Self-updating

Multiple remote repositories would allow companies to generate their own internal versions and distribute more easily, or enable "beta" or "dev" channel repositories.

Maintaining a single class path directory would not be reasonable for dealing with source files. Similarly, it would be nice to be able to add multiple custom library paths -- similar to how class paths are resolved.

Support for wildcard versions would really, really help (especially if semver is respected). You can request library version 1.0.*, allowing for patch releases. This is a great middle-solution between "Give me version 1.0.4 exactly" or "Give me any installed version", and a frequently requested feature for OpenFL

As a cherry on top, it would be nice if libraries could install command aliases, such as lime for haxelib run lime or as3hx for haxelib run as3hx

The best package manager in the world will need updates. I would like to see a library that is self-updated when you run a command similar to haxelib update

@elsassph
Copy link

@elsassph elsassph commented Oct 2, 2017

@clemos haxelib run isn't a problem per se, but it should be extended to other runtimes, and be configurable in haxelib.json.

@elsassph
Copy link

@elsassph elsassph commented Oct 2, 2017

Caching libs is good, using symlinks is bad: someone could mistakenly modify a file and it would affect the cache / other projects.
Libs should be copied locally in the project (unless optional CI optimisation to use cache).

@clemos
Copy link

@clemos clemos commented Oct 3, 2017

@Simn @elsassph I agree it's not so much about about deprecating haxelib run itself, but rather about deprecating run.n in favor of Run.hx ;)

@bablukid
Copy link

@bablukid bablukid commented Oct 3, 2017

The package manager should allow custom fields in haxelib.json
It would be very usefull for librairies which are tied to a specific framework or runtime.
i.e : For externs, we could provide a link to the original library and infos about how to integrate it in your haxe project
i.e : a web framework which loads plugins (haxelibs) that have various assets ( like models, controllers, templates and translation files )

@andyli
Copy link
Member

@andyli andyli commented Oct 13, 2017

I would like to point out that it is important to consider how to handle "native code" (could be compiled ndll, C/C++/JS/... compiled libraries or source files).

One reason is that it is wasteful to download binaries of all architectures. e.g. Lime's ndll folder is 108M, but if you're only using Windows, you actually only need the 5.1M ndll/Windows folder.

We should also have some guidelines regarding to how to store the info of those "native" things, e.g. where is the original source, what param is used when building (use of -PIC/-O3/debug info), any additional dependencies that is not bundled etc.

@kevinresol
Copy link

@kevinresol kevinresol commented Oct 14, 2017

Build tools (hxcpp,hxjava,hxcs) should not rely on haxelib too.
They can be invoked by Context.onAfterGenerate or haxe --run

@Simn Simn closed this Oct 14, 2017
@HaxeFoundation HaxeFoundation locked and limited conversation to collaborators Oct 14, 2017
@Simn
Copy link
Member Author

@Simn Simn commented Oct 14, 2017

Stage 1 has concluded, I'll prepare the discussion part for Monday.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet