Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP agda: Add libraries semi-automatically from package-index #87903

Open
wants to merge 1 commit into
base: master
from

Conversation

@turion
Copy link
Contributor

turion commented May 15, 2020

Motivation for this change

We don't want to keep a manual list of agda packages. This already exists in the form of https://github.com/agda/package-index/, and should be reused as far as automatically possible

This is work in progress. Further things to be done:

  • Find a good way to select the best revisions for each library. Either we fix a rev here in nixpkgs, or there has to be an automatic way to get the newest recorded revision from package-index.
  • Try to build all libraries and mark
  • Metadata
    • Add myself (and @alexarice ?) as maintainers for each package
    • Descriptions and URL
  • Overwrite mechanisms per library
    • Choose a different rev
    • add more maintainers
    • dependencies (those that cannot easily be parsed)
    • Custom files or build phases
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@alexarice

@turion turion mentioned this pull request May 15, 2020
@alexarice
Copy link
Contributor

alexarice commented May 15, 2020

My intuition is that a separate script like cabal2nix would be easier. i.e. a script you run that generates a load of nix files

@veprbl
Copy link
Member

veprbl commented May 15, 2020

@turion This is an interesting approach, but it is not Hydra-friendly as you can see.

@alexarice
Copy link
Contributor

alexarice commented May 15, 2020

I am still not sure how often the package index is updated either, I still think it is better to maintain our own package versions

@turion
Copy link
Contributor Author

turion commented May 16, 2020

@veprbl What is the error? I have no idea how to debug this.

@emilazy
Copy link
Member

emilazy commented May 16, 2020

You can't import, readFile, readDir, ... the result of a derivation (such as fetchTarball) in nixpkgs; that's called (somewhat inaccurately) import-from-derivation and Hydra forbids it. It leads to builds happening at evaluation time, whereas otherwise in the Nix model they're separate: Nix code is evaluated to produce a derivation which is then built.

Basically, you'll have to adopt the approach @alexarice suggested: write a script (possibly in Nix) to fetch the tarball and process its contents, but check the generated code into nixpkgs.

@turion
Copy link
Contributor Author

turion commented May 16, 2020

@emilazy I see, thanks for the explanation. I guess that's the reason why Haskell on nixpkgs has its complicated workflow. Is there any documentation or discussion of this restriction?

@emilazy
Copy link
Member

emilazy commented May 16, 2020

It's avoided because it means that the build graph cannot be determined statically, and you can't answer questions like "what attributes are available in agdaPackages?" without doing arbitrary fetches/builds. Practical implications include not working well with --dry-run and nix-env(1). I'm afraid I don't know a pointer to a specific central discussion on this, but see e.g. #24590 (comment), #11319 (comment), #87592 (comment).

The approach I would suggest is to have a script (in Nix itself, or Python, or even just a bash script if you really want) that downloads the package index and extracts the package names, URLs and hashes to a JSON file; then you can construct agdaPackages from the underlying data with builtins.fromJSON (builtins.readFile ./packages.json).

@turion
Copy link
Contributor Author

turion commented May 16, 2020

@emilazy Thanks again for your advice!

@alexarice Ok, then I guess it's a script. Maybe agda-pkg can be used for that somehow..?

I am still not sure how often the package index is updated either, I still think it is better to maintain our own package versions

Hmmm, package-index seems to be a one-person project run by @jonaprieto, but still it's in the agda organisation, and it seems to receive updates every odd months. Also, since it's open, we can as well send PRs there. Then the non-Nix-using (part of the) Agda community can also profit.

And we in turn profit from the listings of new and updated packages there. I wouldn't underestimate that. If we run our own library set here, we can quickly get out of sync with the rest of the community, and I would very much like to avoid that, because it would mean losing any adoption that nix might win very quickly.

I asked the maintainer of package-index for collaboration here: agda/package-index#20

@alexarice If we mainly pull packages from package-index and add mechanisms for overrides, is that an acceptable solution for you?

@alexarice
Copy link
Contributor

alexarice commented May 16, 2020

I personally don't see how agda-pkg will help, as I believe the main purpose of it is to download the git repos, which nix will do for us

@jonaprieto
Copy link

jonaprieto commented May 16, 2020

I am still not sure how often the package index is updated either, I still think it is better to maintain our own package versions

I update the package-index every once in a while, without any periodicity. Unfortunately, some packages do not have any release, so we keep the last commit seen as their version.

One may think to have one script to keep the package-index updated. It can not take longer. However, I have not found time to do that, to be honest.

It's avoided because it means that the build graph cannot be determined statically, and you can't answer questions like "what attributes are available in agdaPackages?" without doing arbitrary fetches/builds. Practical implications include not working well with --dry-run and nix-env(1). I'm afraid I don't know a pointer to a specific central discussion on this, but see e.g. #24590 (comment), #11319 (comment), #87592 (comment).

The approach I would suggest is to have a script (in Nix itself, or Python, or even just a bash script if you really want) that downloads the package index and extracts the package names, URLs and hashes to a JSON file; then you can construct agdaPackages from the underlying data with builtins.fromJSON (builtins.readFile ./packages.json).

Agda-pkg may help you with that. Run the following commands, each command have their own flags, so check them with --help.

$ pip3 install agda-pkg
$ apkg init
$ apkg list
Library name         Latest version  URL
---------------------------------------------------------------------------------------------------------
agda-base            v0.2            https://github.com/pcapriotti/agda-base.git
agda-categories      v0.1            https://github.com/agda/agda-categories.git.
...
$ apkg info standard-library
{ 'cached': False,
  'default': True,
  'depend': [],
  'description': None,
  'editable': False,
  'fromGit': True,
  'fromIndex': True,
  'fromUrl': False,
  'include': 'src/',
  'index_path': PosixPath('/Users/jonaprieto/.apkg@agda-2.6.1/package-index/src/standard-library/versions/v1.3'),
  'installed': False,
  'keywords': [],
  'library': 'standard-library',
  'license': '',
  'origin': 'https://github.com/agda/agda-stdlib.git',
  'sha': '9f929b4fe28bb7ba74b6b95d01ed0958343f3451',
  'source_path': PosixPath('/Users/jonaprieto/.apkg@agda-2.6.1/package-sources/standard-library@v1.3'),
  'testedWith': [],
  'version': 'v1.3'}

Hmmm, package-index seems to be a one-person project run by @jonaprieto, but still it's in the agda organisation, and it seems to receive updates every odd months. Also, since it's open, we can as well send PRs there. Then the non-Nix-using (part of the) Agda community can also profit.

Indeed.

I personally don't see how agda-pkg will help, as I believe the main purpose of it is to download the git repos, which nix will do for us

Download git repos is one thing. Agda-pkg has more usages. See the documentation https://agda.github.io/agda-pkg/#/?id=usage-manual
If there is something you want to add, please add an issue, or PR.

BTW, someone a few weeks ago added something related to NixOS, you might be interested.

https://agda.github.io/agda-pkg/#/?id=using-with-nix-or-nixos

@alexarice
Copy link
Contributor

alexarice commented May 17, 2020

Ah i did not realise there were more commands, I expect the JSON output could be used

@alexarice alexarice mentioned this pull request May 17, 2020
4 of 10 tasks complete
@alexarice
Copy link
Contributor

alexarice commented May 17, 2020

I've had more of a look at agda-pkg. It does a lot more than I thought it did, and the json output would be very useful, though I am not sure how you would plan on generating build instructions based on the data. For example, there is no information anywhere that ial should be built with make (that needs a patchShebangs according to our current package) and standard-library needs you to run a haskell script.

@turion
Copy link
Contributor Author

turion commented May 17, 2020

though I am not sure how you would plan on generating build instructions based on the data

I can see three ways:

  1. PR to the library itself that gets rid of the make script. Gives the cleanest solution, but is the most effort, may yield friction with the library maintainers, and is often not feasible.
  2. That information needs to be encoded by hand in package-index, but I don't know how. @jonaprieto how would you deal with that situation?
  3. Allow to override the buildPhase in nixpkgs.

I think there are not enough Agda libraries out there in order to try and detect the build strategy automatically. Probably it's going to be option 3., since it's easiest and also quite stable and versatile.

@alexarice
Copy link
Contributor

alexarice commented May 17, 2020

Once you do option 3 I don't see the point of having automatic package generation

@jonaprieto
Copy link

jonaprieto commented May 18, 2020

3. Allow to override the buildPhase in nixpkgs.

I'll choose this approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.