php: add new Composer builder

[drupol]

Introducing a new builder for PHP Composer-Based Applications

Ensuring reproducibility and stability of outputs is required even when using an external dependency manager (Composer). As such, we've explored several possible solutions.

1. The first attempt of this PR was using a Fixed Output Derivation strategy, using the Composer cache. Although this solution rectified numerous issues, it was not entirely secure, as it was heavily dependent on Composer. If Composer implemented even a minor structural alteration in the cache structure, it could break all derivations based on this builder. As a result, we concluded that this solution wasn't ideal.

2. Our second attempt involved the use of fossar/composition-c4. This project reads the composer.lock file and constructs a local Composer repository, which is then used to install the dependencies. However, as Import From Derivation (IFD) is not permitted in nixpkgs, this approach didn't serve as a feasible solution. Moreover, path type repositories are currently unsupported, adding another roadblock.

3. A proof-of-concept led to the creation of our last and final attempt; a custom Composer plugin, nix-community/composer-local-repo-plugin providing a new command. It mirrors the functionalities of fossar/composition-c4 and is written in PHP. It relies entirely on Composer to create a local Composer repository, thus emerging as a solid solution.

Although this plugin is currently hosted under my own name, there is a consideration to host it elsewhere, a decision yet to be made.

Here's a diagram overview on how the new builder is working

This PR includes the following changes:

• Introduces two new helpers (mkDerivation wrappers)
  • php.buildComposerProject
  • php.mkComposerRepository
• Introduces two new hooks
  • php.composerHooks.composerRepositoryHook
  • php.composerHooks.composerInstallHook
• Documentation

This new helpers significantly simplifies the process of integrating applications such as Symfony App, Wordpress or even Drupal into Nix, streamlining development and deployment.

Below is a practical example demonstrating how to efficiently package Drupal using this approach:

Or magento2:

Todo

• Add a parameter that enable or disable the flag --no-dev

Upcoming PRs

• Add a new helper that parse composer.json for extensions and instantiate the adequate php attribute with these extensions enabled. This has already been done in https://github.com/loophp/nix-shell/, we just need to formalize it in here.

[n0emis]

I've been testing this builder with some packages and find it very intuitive to use and am hoping to get it included soon. 👍🏼

[RaitoBezarius]

@drupol except if you express strong rejection, I suggest that we move forward whatever is left to do to another PR and I can merge this now.

[drupol]

Give me an extra day so I can make last minor changes please :)

[etu]

I'm also thinking about commit messages.

Rather than the prefixes builder: and feat: maybe you should just put php:

I think it goes more in line with the contribution guidelines that way

EDIT: We can also just squash merge it using GitHub.

[RaitoBezarius]

Actually, yes, we never have feat: in our commit messages in nixpkgs, all prefixes should ideally follow a semantic path to the contents: build-support/php is a good example.

[drupol]

Let's squash all of these commits during the merge, WDYT?

[RaitoBezarius]

Let's squash all of these commits during the merge, WDYT?

I disagree, this is valuable history which is not that bad in terms of atomicity, losing it does not really make sense or bring any value IMHO.

[sinavir]

Hello, does this PR handle git (vcs ?) dependencies as specified here ?

If not may be it could be stated in the docs !

I'm trying to package the above software and the tooling introduced by this PR gives me this :

Failed to clone the git@github.com:LycheeOrg/phpstan-lychee.git repository, try running in interactive mode so that you can enter your GitHub credentials

In Filesystem.php line 255:

  /homeless-shelter/.cache/composer/vcs does not exist and could not be creat
  ed.

(the full nix expression is here : https://github.com/sinavir/sinavir/blob/lychee/lychee-nix/lychee.nix)

Nevertheless, this PR seems amazing ! Thanks for your work !

(PS: I'm not very familiar with composer so it may be just me not understanding how things are working)

[drupol]

Hello, does this PR handle git (vcs ?) dependencies as specified here ?

If not may be it could be stated in the docs !

I'm trying to package the above software and the tooling introduced by this PR gives me this :

Failed to clone the git@github.com:LycheeOrg/phpstan-lychee.git repository, try running in interactive mode so that you can enter your GitHub credentials

In Filesystem.php line 255:

  /homeless-shelter/.cache/composer/vcs does not exist and could not be creat
  ed.

(the full nix expression is here : https://github.com/sinavir/sinavir/blob/lychee/lychee-nix/lychee.nix)

Nevertheless, this PR seems amazing ! Thanks for your work !

(PS: I'm not very familiar with composer so it may be just me not understanding how things are working)

I tried locally and I get:

❯ composer i
No composer.lock file present. Updating dependencies to latest instead of installing from lock file. See https://getcomposer.org/install for more information.
Loading composer repositories with package information
GitHub API limit (60 calls/hr) is exhausted, could not fetch https://api.github.com/repos/LycheeOrg/php-exif/contents/composer.json?ref=c78ec611bea970d61cfbf721dc4b5abb33437901. Create a GitHub OAuth token to go over the API rate limit. You can also wait until 2023-09-13 16:02:22 for the rate limit to reset.

When working with _public_ GitHub repositories only, head to https://github.com/settings/tokens/new?scopes=&description=Composer+on+x13+2023-09-13+1502 to retrieve a token.
This token will have read-only permission for public information only.
When you need to access _private_ GitHub repositories as well, go to https://github.com/settings/tokens/new?scopes=repo&description=Composer+on+x13+2023-09-13+1502
Note that such tokens have broad read/write permissions on your behalf, even if not needed by Composer.
Tokens will be stored in plain text in "/home/pol/.config/composer/auth.json" for future use by Composer.
For additional information, check https://getcomposer.org/doc/articles/authentication-for-private-packages.md#github-oauth
Token (hidden):

The new builder should support this kind of repo, but I have the feeling that you reached the limit for accessing that repo, that's why you get the error. Maybe this is something we could improve at some point (I have no clue how yet).

[drupol]

Let's squash all of these commits during the merge, WDYT?

I disagree, this is valuable history which is not that bad in terms of atomicity, losing it does not really make sense or bring any value IMHO.

Ok going to rewrite all of them properly.

[drupol]

@GrahamcOfBorg build phpPackages.composer

[sinavir]

The new builder should support this kind of repo, but I have the feeling that you reached the limit for accessing that repo, that's why you get the error. Maybe this is something we could improve at some point (I have no clue how yet).

Ok, Thank you for the information. I don't have the throttling message and composer i is working for me inside pure nix-shell created using the (failing) derivation I linked above. I will try to debug this on my side.

[drupol]

Well well well.. @RaitoBezarius or @etu ?

[yu-re-ka]

I'm a bit late on this, but does the buildComposerProject also have a guard against forgetting to update the FOD-hash? In yarn and Cargo builders, we include the lockfile in the fixed-output derivation, and then check that the lockfile in the deps FOD is the same as in src when building.

[drupol]

I don't think we have that ... yet?

[lorenzleutgeb]

@drupol Regarding #237307 we wanted to execute unit and possibly integration tests as part of the derivation created via buildComposerProject. It seems that checkPhase is exposed, but PHPUnit tests have to run after composer install, which is in installPhase. Probably buildComposerProject should also expose installCheckPhase. We also checked some packages that use buildComposerProject and could not find one that actually runs tests. Happy to have your comment/explanation on how you thought that testing should be achieved with this mkDerivation wrapper. Thanks!

[drupol]

What do you think of this? Could you test please? #261429

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/contribute-to-the-nix-2023-recap/37484/8