Binary crates, toolchain management and use #775

mosteo · 2021-07-26T10:08:24Z

This PR implements a variety of changes to support the use of binary compilers packaged through Alire:

New binary crates.
provides property to allow mixed gnat/gnat_xxx dependencies.
- Changes in the solver and related types to use this property properly.
- Changes in the solver to prefer locally installed binary crates to remote available ones.
alr toolchain to inspect/[un]install/select toolchains.

Still missing:

Passing existing tests.
New tests.
Scrub the code for leftover comments / self-review.

Documentation might be a bit sparse yet, and require some tweaking.

@Fabien-Chouteau , now the toolchain selection assistant is run the first time a valid alire folder is required. We might disable this entirely, or run it only if there is no gnat/gprbuild in the PATH, or something else if you see fit.

Essentially, we have a second location for dependencies at ${ALR_CONFIG}/cache. In following PRs the releases therein will be available during dependency resolution, to be reused. `alr install` for manual crate installation Installed crates are used by the solver Autodiscover installed releases Show differences due to changed install status Install: implement release removal

Menu for choosing among multiple options Show actual compiler versions to the user Install and remember the user choice User_Input: Take a possible Ctrl-D into account

New, more flexible provides type WIP: going back to provides as list of deps All but loading of "provides" Loading of provides Installed provides are used during solving They're not yet used for regular dependencies Display provider in the `alr with` variants Prevent solving of conflicting dependencies If two releases provide the same crate, they cannot be simultaneously in a solution. solver tweaks for provides + externals Improvements to solution space by reusing releases When adding a release to a solution, check whether other dependencies can be satisfied by this same release. Not a big deal, could be disabled if too costly in the future.

Insert explicit dependency on the configured gnat Do not explicitly insert dependency on toolchain Instead, we simply coerce when given a generic gnat dependency into the configured one. This is less intrusive as the user does not see unexpected dependencies on the compiler in the solution (unless explicitly requested). Extra info in `alr install` list Generalized assistant for all tools in toolchain

For now this is restricted to a single alternate crate name. By necessity, the equivalence has to be to the same version, whereas provides for regular releases are more general as they can rely on precise versions which are known in advance.

This is logical because these are always available in all configurations. But the important benefit is that the solver can use them in preference to a remote native compiler.

src/alire/alire-utils-user_input.adb

doc/catalog-format-spec.md

testsuite/tests/solver/compiler-mixing/test.py

mosteo · 2021-07-29T16:41:03Z

Hi @Fabien-Chouteau . I still have to see what's going on with Windows, but this one is ready for review. I apologize in advance because on top of being a large PR in itself, I had to do a couple of refactors towards the end to break circularities that mud things even more. I'll try to compartmentalize things better next time 😓

Fabien-Chouteau · 2021-07-30T08:41:07Z

doc/toolchains.md

+   1. A cross-compiler from the index, that will be deployed with the rest of dependencies.
+
+- Solution with dependencies on a target-specific GNAT (`gnat_native`, `gnat_ricv_elf`,
+  etc.): The `alr` solver will provide one compiler in the environment, applying the


Is it possible to have both an explicit dependency on gnat_native and gnat_riscv64_elf? It should be.

Simultaneously in a crate? How's that supposed to work?

I have one video game crate (shoot_n_loot) that builds both the embedded version of the game and a native simulator to play it on the computer. But maybe that's a very bad idea ^^

I actually was worried about this use case but I guess I didn't traslate it properly in our talks. This is the one where I saw your idea on conditional dependencies on something provided via command line/dependent crate configuration could be useful:

alr build target:=native # arbitrary name, not predefined (or defined in the configuration section of the crate)

and then we can have in the toml:

[[depends-on."case(crate.target)".native]] gnat_native = "*" [[depends-on."case(crate.target)".riscv]] gnat_riscv_elf = "*"

There should be a default somehow for automated builds though.

For a later date obviously...

Otherwise, I guess we can do an exception about the gnat crate not being conflicting among compilers. But something nags me at that being unkind to people not interested in all targets.

Let's keep it simple for now and only allow one compiler.

I have one other use case.

A cross project that uses native tools to generate code, for instance .

Right, that is a legitimate situation that we ought to support. I'll try to come up with a solution in a separate PR.

Addressed in #780

doc/toolchains.md

Fabien-Chouteau · 2021-07-30T10:04:39Z

src/alire/alire-dependencies-states.ads

+   function Is_Provided (This : State) return Boolean;
+   --  True when the release name is different from the dependency crate
+
+   function Is_Shared (This : State) return Boolean;


Should it be Is_Installed?

I started using the idea of 'shared releases' internally, as these are just regular releases in a shared location. But it can be changed if confusing (likewise then for the Alire.Shared package).

It will depend on how and if we implement an alr install I guess. We can keep shared for now, up to you.

My current mood is to avoid install entirely. I was thinking of using alr share for non-toolchain crates shared in the configuration, and alr export for the gprinstall-in-common-prefix idea. So we convey that exported crates are something the user wants to use out of Alire. But I'm easy to sway on these naming things.

For now if you don't mind I'll leave it as is then.

If the user wants to make some application available system-wide, I would say the usual keyword used by package managers and build systems is install instead of export. export sounds like an action to extract data from an application.

In this case we would be "extracting" binaries from the Alire ecosystem. But I see your point. alr share for shared dependencies across workspaces and alr install for deploying crates to a common prefix can also work for me.

Another possibility is to have it under get, e.g., alr get --install. The operation in itself is essentially an alr get + alr build + gprinstall.

src/alire/alire-releases.adb

In particular, the avoidance of downloading a compiler for a generic gnat dependency. This is tested in testsuite/tests/solver/compiler-installed

mosteo · 2021-08-17T18:30:31Z

Hi @Fabien-Chouteau , with #780 and #781 in the pipeline I think we are ready feature-wise for this one. There is the matter of whether to disable sha256, and the reworded explanation in toolchains.md. I can take another stab at it if you want, unless you're already on it.

Fabien-Chouteau · 2021-08-18T07:56:03Z

Thanks @mosteo

I can take another stab at it if you want, unless you're already on it.
Unfortunately no, I didn't take the time to work on this.

Maybe we can do a release candidate and ask for feedback. Based on the feedback we could be able to improve the doc...

mosteo · 2021-08-18T08:38:56Z

OK then, in fact I should update the doc in #780 at least to match the current situation.

mosteo and others added 11 commits July 22, 2021 19:16

Implement sha256 and hash use test (#765)

aa75b8b

Implement binary origins and their deployment

cb35bd5

Toolchain configuration assistant

ed78718

Menu for choosing among multiple options Show actual compiler versions to the user Install and remember the user choice User_Input: Take a possible Ctrl-D into account

Rename install command to toolchain

b2ca040

Implement "provides" for external detectors

d13bdd9

For now this is restricted to a single alternate crate name. By necessity, the equivalence has to be to the same version, whereas provides for regular releases are more general as they can rely on precise versions which are known in advance.

Tweaks for new GNAT crate names

05007ad

Solver: reuse known GNAT on generic gnat= dep

6a960d6

Integrate external gnats in shared releases

f6a7fdc

This is logical because these are always available in all configurations. But the important benefit is that the solver can use them in preference to a remote native compiler.

mosteo force-pushed the feat/toolchains branch from 3cb0bcc to d151a6c Compare July 26, 2021 10:27

github-actions bot reviewed Jul 26, 2021

View reviewed changes

src/alire/alire-utils-user_input.adb Outdated Show resolved Hide resolved

doc/catalog-format-spec.md Outdated Show resolved Hide resolved

Documentation in doc/, alr help

7096188

mosteo force-pushed the feat/toolchains branch 2 times, most recently from d27e2b5 to 4eb06cb Compare July 27, 2021 08:28

Fixed circularity, Config.Edit is now untangled

07b7b93

mosteo force-pushed the feat/toolchains branch 5 times, most recently from ff45b3f to 3f4db3e Compare July 27, 2021 11:34

Tweaks to pass existing tests

33620d6

mosteo force-pushed the feat/toolchains branch 2 times, most recently from 655e4f9 to d10355d Compare July 28, 2021 16:50

Self-review fixes and clean-up

c7bef0f

mosteo force-pushed the feat/toolchains branch 2 times, most recently from 5b084c1 to 8462f71 Compare July 29, 2021 12:42

github-actions bot reviewed Jul 29, 2021

View reviewed changes

testsuite/tests/solver/compiler-mixing/test.py Outdated Show resolved Hide resolved

mosteo force-pushed the feat/toolchains branch from 8462f71 to ce86d99 Compare July 29, 2021 12:45

mosteo force-pushed the feat/toolchains branch 3 times, most recently from 8353758 to 56639f5 Compare July 29, 2021 16:38

New tests for toolchains and related features

56639f5

mosteo marked this pull request as ready for review July 29, 2021 16:41

mosteo requested a review from Fabien-Chouteau July 29, 2021 16:41

mosteo force-pushed the feat/toolchains branch 4 times, most recently from 57ed3a5 to 78c87fb Compare July 30, 2021 10:20

Windows-specific testsuite fixes

f8c7ebe

mosteo force-pushed the feat/toolchains branch from 78c87fb to f8c7ebe Compare July 30, 2021 10:33

Fabien-Chouteau reviewed Jul 30, 2021

View reviewed changes

mosteo mentioned this pull request Jul 30, 2021

muntsos_blinky release 1.0.0 alire-project/alire-index#304

Closed

mosteo force-pushed the feat/toolchains branch 2 times, most recently from 1710baa to 8cca83d Compare July 31, 2021 12:34

github-actions bot reviewed Jul 31, 2021

View reviewed changes

src/alire/alire-releases.adb Outdated Show resolved Hide resolved

Review fixes

328e051

In particular, the avoidance of downloading a compiler for a generic gnat dependency. This is tested in testsuite/tests/solver/compiler-installed

mosteo force-pushed the feat/toolchains branch from 8cca83d to 328e051 Compare July 31, 2021 12:38

Merge remote-tracking branch 'alire/master' into feat/toolchains

bde643d

mosteo mentioned this pull request Aug 5, 2021

Releases providing the same 3rd crate no longer implicitly conflict #780

Merged

Merge remote-tracking branch 'alire/master' into feat/toolchains

0f411a4

mosteo changed the base branch from master to merging/toolchains August 18, 2021 10:28

mosteo merged commit d6ea245 into merging/toolchains Aug 18, 2021

mosteo deleted the feat/toolchains branch August 18, 2021 10:28

Binary crates, toolchain management and use #775

Binary crates, toolchain management and use #775

mosteo commented Jul 26, 2021 •

edited

mosteo commented Jul 29, 2021

Fabien-Chouteau Jul 30, 2021

mosteo Jul 30, 2021

Fabien-Chouteau Jul 30, 2021

mosteo Jul 31, 2021

Fabien-Chouteau Aug 2, 2021

Fabien-Chouteau Aug 3, 2021

mosteo Aug 3, 2021

mosteo Aug 5, 2021

Fabien-Chouteau Jul 30, 2021

mosteo Jul 30, 2021

Fabien-Chouteau Jul 30, 2021

mosteo Jul 31, 2021

onox Jul 31, 2021

mosteo Aug 5, 2021

mosteo commented Aug 17, 2021

Fabien-Chouteau commented Aug 18, 2021

mosteo commented Aug 18, 2021

Binary crates, toolchain management and use #775

Binary crates, toolchain management and use #775

Conversation

mosteo commented Jul 26, 2021 • edited

mosteo commented Jul 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mosteo commented Aug 17, 2021

Fabien-Chouteau commented Aug 18, 2021

mosteo commented Aug 18, 2021

mosteo commented Jul 26, 2021 •

edited