Allow Composer packages to ship their own recipes

[dkarlovi]

If you're a package developer, you might want to opt-in to being Flex compatible. This way, you're shipping the recipe with the package, maintaining it there (each version might be different, but as they're together, they're in sync). Without this, you maintain your package in your own repo, but need to maintain a part of it somewhere else (in the recipes* repos). If you push a major release, you might have to wait for someone else to approve the PR, etc.

You could allow for packages to ship their own Symfony recipes, maybe directly inside composer.json? It could work like this:

1. developer Dave decides to create a new Symfony bundle
2. he develops it completely and adds composer.json to it
3. he wishes to be compatible with Symfony Flex too so he adds a block to composer.json with the recipe
4. packagist.org sees that the package has additional information and notifies Flex API server about it, this could abstracted with tags in composer.json and supporting tag subscription notifications in packagist.org
5. user installs the bundle, gets the benefits of Flex without Dave pushing his recipes to recipes* repos

This might be already possible somehow, but I didn't find a reference to it. This would also mean recipes-contrib would become (mostly) obsolete because maintainers would be able to ship their own recipes, the only use case for contrib would be adding recipes to bundles where you might not be able to add recipes directly, for example with unmaintained or uncooperative projects. Recipes repo would stay in place mostly for curation purposes (as in, enumerating quality packages and providing aliases for them).

[javiereguiluz]

@dkarlovi I'm afraid this idea has been rejected before because Symfony Flex needs a central repository in order to implement some of the features that are planned for the coming months.

I understand this may be a bummer for you, but it's common for Symfony developers to use "central repositories". They use GitHub.com as the central repository of their public code and Packagist.org as the central repository to define Composer packages.

[dkarlovi]

@javiereguiluz the difference is, they control their own additions to the central repository, it doesn't go through some 3rd party.

[dkarlovi]

Also, you still have the central repo, as stated in 4), you just don't need to go through the recipes repo.

[javiereguiluz]

I disagree with you about the 3rd party comment. For your code you rely on a private 3rd party called GitHub.com and for the Composer packages you rely on a private 3rd party called Packagist.org (this one has open sourced some parts of it). You can get banned by them and they have Terms of Use, so your code/package can be rejected by them.

The idea is to apply the same rules for symfony/recipes-contrib ... everything will be approved, except some clearly illegal things, etc.

[dkarlovi]

@javiereguiluz you misunderstood my "3rd party" comment, let me clarify: I'm not talking about "being free of 3rd party services" nor imply Github or Packagist or somehow free for me to use or abuse.

My point is, having to go through recipes-contrib seems awkward for maintainers and might hinder common workflows.

Here are some scenarios:

1. say you're starting out making a package which is interesting for you, and might be interesting to some (for example, I have experience with Docker+Symfony so I created a repo symfony-flex-docker), you'd like to play around with composing a Symfony app with your addition. How do you proceed: first thing you need to do is do a PR on some completely other repo (recipes-contrib)? But you don't have any code yet, let alone being versioned (as in, package version). You might also not want to pledge being a maintainer of this package this early (or at all), but you still need to "advertise it" (this is moot as you need to do the same on Packagist, but worth noting).
2. say you've bumped a minor version of your package. Do you need to go to recipes-contrib and bump it there too? Each time? What happens until you do (or if you never do), does your minor version not work with Flex from now on?
3. say you're doing a major release, you publish your changes, do a PR on recipes and everything. For some reason, your PR is not merged for a week (or two, it wouldn't be unheard of). In a Flex-driven future Symfony world, this means your release is delayed for that one (two) weeks by somebody else.

I'm not knocking the "central repository" part, I'm against the inclusion mechanism. If everything will be approved, why not approve it automatically, like Packagist does? Why not let maintainers have a mono-repo like Symfony itself does? Having a separate repo to do this feels like... paperwork.

Thanks for your answer.

[Pierstoval]

@dkarlovi There is the same kind of integration workflow for debian packages in the official repos, and there's no complaint about this, because it's the best way to have control over flex packages.

Bundling recipes in packages themselves would be asking something to composer, which is not the case of Flex which is just retrieving the package naturally and adds some logic afterwards if there is a recipe corresponding to this specific package. That's completely different.

The goal with the Flex repos is to provide "Official" and "Contributors based" repositories to find flex packages.

Maybe @fabpot will one day propose users a way to store their own Flex repositories, by open-sourcing a flex server project, like you can have your own "private packagist server", but users will need to be aware that unofficial flex repositories will offer no guarantee of the quality & compatibility official ones provide.

[dkarlovi]

@Pierstoval

there's no complaint about this

There's been extreme complaints about the fact each distro needs to re-package their own stuff (and, by which, reinvent the wheel, very NIH). That's the reason Snap or Flatpak was created (again, competing standards).

Also, it's common for system software to provide their own packaging spec (example: PHP, Sphinx Search) which might be used directly or extended by the distro.

Bundling recipes in packages themselves would be asking something to composer, which is not the case of Flex which is just retrieving the package naturally and adds some logic afterwards if there is a recipe corresponding to this specific package.

Flex is, as it stands, completely dependant on Composer because it's literally a Composer plugin. Future direction might change that, of course, but currently it's exactly like that.

The goal with the Flex repos is to provide "Official" and "Contributors based" repositories to find flex packages.

That's not brought into question, official repo curation would be great to have, I'm talking specifically about recipes-contrib and the way to get Flex to recognize your package as supported.

[dkarlovi]

To recap, everything stays as is, only change is Flex checks for recipe in composer.json of the installed package too, not only recipes* repo. If it's not there, everything works as it currently does.

[Pierstoval]

About complaints, I agree I came to fast by saying there were none. The issue here is that packages need lots of validation to be integrated into the official repos, which makes the packages extremeley reliable, most of the time.

For flex being a composer plugin, I never said that it was not related to composer. Flex is catching composer installations, requirements and removals, to ask Flex recipe if the composer package is available with Flex (= it has a flex recipe, and/or it is a symfony bundle that can be automatically added to the kernel if there is no recipe). Flex is not a package manager, it's a package installation/configuration automation system, that's why it's different than composer itself

[chalasr]

Composer and github gives packages, while flex integrates them to your application. The formers just put things beside your application, the latter can actually alter your application. That's not the same target, not the same validity checks to perform.
Removing the recipes-contrib repository in favor of a composer config on the package itself (keeping the allow-contrib behavior as is => if the recipe is not in the main recipes repo, then it's a contrib) could be nice, but that would prevent us to ensure/challenge the recipes validity (the way it alters the requesting apps) .

I think it's legit to keep control over contrib recipes even if they are almost all accepted, they are all reviewed beforehand so you can't end up with weird things in your application, that is part of the flex responsibility.

[dkarlovi]

@Pierstoval

packages need lots of validation to be integrated into the official repos, which makes the packages extremeley reliable, most of the time.

I wouldn't otherwise comment this part further, but since you've said "official repos", "validation" and "extremely reliable" (specifically with regards to Debian), you've named all the keywords required. :)
(TLDR: Debian maintainer, during package validation, disabled a key part of upstream he's packaging, crippling it and introducing an amazingly high-impact vulnerability, I'm not saying this is bound to happen with Flex in any way, it's just amusing)

Flex is not a package manager

I understand what Flex is supposed to be (on technical level, at least). The only difference I've proposed here is that it tries to fetch the recipe from the package being installed too.

To clarify again: I see great value in Flex even now (that's the reason I'm still commenting on a closed issue), This especially goes for proposed "packs" (which, in my mind, are Composer metapackages which only list the exact dependencies and optionally their exact configuration).

For example, ApiPlatform could become a pack with exact versions of 3rd-party packages required and (optionally) their configuration. Let's say it needs to configure FOSOauthServerBundle in a specific way to have it work, but OauthBundle doesn't have a recipe yet. Do they submit their version of the bundle's recipe? It's not really generic so that's probably wrong.

A better way would be to create a pack which would:

1. list the dependecies (probably exact versions)
2. ship configuration for FOSRestBundle (even though it has an entry in recipes and/or bundle itself)
3. rely on configuration of other components coming from their packages / recipes repo

This way, you get to have official packs, platform vendors such as ApiPlatform or Sylius can drop their "standard editions" in favor of packs, maintainers can ship their own (default) configurations, it seems really flexible to me.

[dkarlovi]

@chalasr

that would prevent us to ensure/challenge the recipes validity

You're trusting developers of the code being executed to know how to write the code in question, give them the benefit of the doubt they'll be able to recipe it. :) See my Debian link previous comment.

In my mind, there's no issue here:

1. end user (developer) decides to install (and does) the bundle
2. Composer installs bundle
3. bundle declares it supports Flex and how it should be configured
4. Flex configures the bundle as told

If by the end your app is broken, that's no more the issue of Flex than it is of Composer (obviously, barring bugs).

Anyway, thanks for your answers, agree to disagree I guess.

[sstok]

Note that as Flex is just a Composer plug-in (and convention) you can actually create your own plug-in for the more simpler tasks (copying files for example) ;)

Flex is not doing more then you were already were able to do, it just makes it more "standard" and moderated to provide some level of quality.

[francoispluchino]

@sstok Sure enough, we could do it, but it's not standard, everyone will create his own plugin for Composer, and a lot of code will be duplicated unnecessarily.

Knowing that in addition, a request to the server https://symfony.sh is done for each package, which is totally useless for a private package, and in addition, an unnecessary load for the server.

Another problem with private packages: their are not published on Packagist, and consequently, the recipe will not be accepted in official or contrib because it is a pre-requisite to be validated.

I accept the argument of security even if I do not agree with it, And the arguments of @dkarlovi explains it very well, but it concerns only the public recipes, not the private recipes. However, we can do like for recipes-contrib, that is, put a key in the composer.json file to accept the use of recipes in packages, And why not, have a default value, and an map of exception for each package (see issue #71).

@javiereguiluz For a private package, you do not use Packagist with Composer, but you use a Satis server or a VCS Repository that requires no server but is slower.

@javiereguiluz, @sstok, @chalasr, @Pierstoval At this stage, you will answer me that it will soon be possible to set up a private server to solve this problem. However I would reply that it would be better to have a flexible approach, which would be a plus for the developers that we are. Composer, Yarn, NPM, Bower do it very well without private server required via GIT SSH (NPM has since added the ability to save private packages, but it pays off and optional).

So for our private packages with Flex, it would give:

1. Recipes on https://symfony.sh
2. Recipes contrib on https://symfony.sh (require approval in composer.json file)
3. Recipes on private server (require approval in composer.json file)
4. Recipes in each package (require approval in composer.json file)

Knowing that 1 and 2 are already implemented in Flex.

I am pleased to note that Symfony is heavily and increasingly inspired by what is best in the javascript world, and I can only adhere to it.

And to get there, you've created this tool, Awesome! But what astonishes me is that the tool is limited, not because it is young, but by choice (In relation to the explanation given, but I'm probably wrong, and I hope ;)).

However, adding flexibility isn't the fundamental principle of this Composer plugin???

I sincerely hope to reopen the debate.

[francoispluchino]

In response to @chalasr 's comment:

I think it's legit to keep control over contrib recipes even if they are almost all accepted, they are all reviewed beforehand so you can't end up with weird things in your application, that is part of the flex responsibility.

@chalasr Sorry, but I don't agree with you, and I don't see why Symfony Flex must guarantee the "security" of the application. I see rather Symfony Flex as a tool for automating tasks plugged to Composer, and I'll try to explain my point of view below.

Symfony Flex is a pure plugin for Composer, It uses the event system of Composer (POST_PACKAGE_INSTALL, POST_PACKAGE_UPDATE, POST_PACKAGE_UNINSTALL, POST_CREATE_PROJECT_CMD, POST_INSTALL_CMD, POST_UPDATE_CMD) to manipulate the project.

Now, Imagine that Jordi and Nils had the same perspective for Composer, that is, ensure that packages downloaded from Packagist/Satis/VCS Repository Drivers, must will be validated by the Composer's team, because everything emanating from the Composer must be guaranteed: the plugin system (composer/composer#2013) would never have been created, and consequently, this plugin would never have existed either, as well as many other plugins like for example fxp/composer-asset-plugin.

I'll take the example of this plugin, because it uses the events of the solver SAT (among others) to manipulate the packages before and after the resolution of the dependencies. This plugin could not exist without composer/composer#2013 and composer/composer#3171 (you can find the first negative feedback from Nils for this feature in the issue composer/composer#3116, with the same reluctances).

In the end, this does not prevent in any way all developers of Oro CRM/Oro Platform, Yii Framework and other, to use this plugin despite an advanced extension of Composer, not originally planned by its creators. They use this plugin because it is a useful feature and they trust the project developer or directly to the developers of the packages.

To return to Symfony Flex, The basis of this plugin is great, as is the system of shared recipes (offcial and contrib). But just like Composer has these beginnings, this plugin should set up a system to extend it, with why not, an activation by the developers in the project to avoid the "weird things" without its consent.

To answer of the suggestion of @sstok in his comment:

Note that as Flex is just a Composer plug-in (and convention) you can actually create your own plug-in for the more simpler tasks (copying files for example) ;)

Flex is not doing more then you were already were able to do, it just makes it more "standard" and moderated to provide some level of quality.

Yes it is indeed the case, but it is frankly damaging to have to reinvent the wheel, and copy code unnecessarily, without being able to execute our plugin before or after your plugin. This vision is therefore not a "standard". Like indicated in the issue #71, I had already proposed the possibility to configure automatically the bundles (self-configuration), but this had been refused at the beginning of Symfony 2.x. Of course, I didn't wait Symfony Flex to create this feature (idem for Oro), and so I use a Composer plugin to do it. However why my plugin is it not public? Simply because it isn't "standard" to Symfony, but only for my project, that it isn't public. Of course, I could go out the base, and put it in open-source, but I considered at the time not having the time to support this kind of tool, and now, Symfony Flex could be this base.

As I said at the beginning of this comment, I see much more Symfony Flex as a great automation tool to standardize the project configuration system, than a simple tool dedicated to Symfony, despite what you say, and even if you have added directly into the plugin a special configurator for the Symfony Bundles (Symfony\Flex\Configurator\BundlesConfigurator).

But for this, @fabpot would have to move in this direction, which seems to be the case with his comment in the issue #53:

the recipes repository is not just for Symfony bundles, but for any Composer packages (like PHPUnit, Twig or Swiftmailer)

Given that Composer is able to have plugins depending on other plugins, It would be great to allow a plugin to listen to events from your plugin, for example, be able to add a new configurator, or build a system for managing recipe servers.

Symfony have the the power to carry this kind of tool, we only have to see what you're doing with all Symfony components used by many open-source or proprietary projects, Or your participation in PSR to be convinced. I think that a "standardized" and extensible automation tool would be great, but for that, we must accept more flexibility and openness to ideas that we had not especially had at the outset, so please, reopen this issue of @dkarlovi to continue to debate.

To conclude, simplify, standardize the project configuration for any type of package, As did Composer for the dependency management, or PSR for the autoloading, the logger, the DI, the cache, the style etc...

Is not this a great perspective for a tool whose main purpose is to help developers to configure their project?

I know, the question is much wider than the original question (and my comment is really very long :)), but ultimately, the problem is the same: How far does Symfony Flex want to help developers for the configuration of their projects?

[sstok]

Given that Composer is able to have plugins depending on other plugins, It would be great to allow a plugin to listen to events from your plugin, for example, be able to add a new configurator, or build a system for managing recipe servers.

OK, now you got me interested 😄

[chalasr]

@francoispluchino Fair enough.
👍 for packages managing their own recipes (contrib only).

[Pierstoval]

Given that Composer is able to have plugins depending on other plugins, It would be great to allow a plugin to listen to events from your plugin, for example, be able to add a new configurator, or build a system for managing recipe servers.

All this extension system could be done by forking Flex and changing some urls, or publish your own plugin based on Flex and override some stuff in Flex. First it needs reverse engineering about the result from https://symfony.sh which is not open-source for now, and I hope @fabpot may propose to publish it one day, even if it's planned for the end of the year, because actually just having a personal Flex server does not seem to be a big deal (it may is, obviously, I'm exaggerating, but the goal of this server is to use Packagist's API & recipes repository to find packages).

[francoispluchino]

@Pierstoval

All this extension system could be done by forking Flex and changing some urls, or publish your own plugin based on Flex and override some stuff in Flex.

A fork, a fork a fork... and always a fork! :) The principle of this proposal is precisely to avoid fork a tool for everyone to have its tool, but have a common foundation. And currently, we cannot extend Flex given that we would have to dispatch events (among others).

First it needs reverse engineering about the result from https://symfony.sh which is not open-source for now

It's not necessary for private recipes, but to create her own Symfony.sh server, yes, Even if it is not really necessary to do reverse engineering on the server APIs, but just look at what Flex expects, it's really simple.

and I hope @fabpot may propose to publish it one day, even if it's planned for the end of the year, because actually just having a personal Flex server does not seem to be a big deal

That would be a good thing, but in the current state of Flex, it is not possible (see this comment of the issue #71).

(it may is, obviously, I'm exaggerating, but the goal of this server is to use Packagist's API & recipes repository to find packages).

Yes I confirm ;) And Composer does not need Packagist to work. We can disable Packagist and only work with a Satis private server (or other), see several private servers, but also with the VCS Repositories (Package repositories directly added to the file composer.json). In no case Packagist is a prerequisite to Compose to execute, but simply the official repository avoiding to manually add the package repositories (and removing analysis of versions). Therefore, Symfony Flex should take this feature into account, in order to work with packages not referenced on Packagist, ie private packages on Satis or VCS Repository.

In the current state of progress of this plugin, https://symfony.sh is used only to retrieve package aliases, and recipes. The recipes with their aliases on https://symfony.sh for packages referenced on Packagist is a very good thing, but this is not the case for private packages, especially since aliases are not required to get recipes. In this regard, the question of fair access to such aliases, as well as potential conflicts with trademarks identifying the use of their names in aliases without authorization would be interesting, but it is another debate :).

To conclude, I remain doubtful, and I still do not understand your reservations. Do not I understand you correctly? Or, Do you simply agree on the principle of adding the possibility of having recipes in the package, but also on private servers, while adding the possibility of extending the plugin with new configurators, but that the Symfony team does not want to spend time for this type of functionality but wants external contributions?

[fabpot]

FYI, pull requests on the contrib repository are now entirely managed by community (and merged automatically), see https://github.com/symfony/recipes-contrib/blob/master/README.rst for the exact rules.

[francoispluchino]

@fabpot Thank you for these new informations about the public packages.

Are the owners (or organization users) notified when an external user pushes a pull request in the recipes-contrib repository?

However, the problem is still present for private packages.

[aurimasniekis]

If anyone is interested in this topic I tried to reverse engineer symfony/flex API and create proxy server/application to allow ship own recipes https://github.com/aurimasniekis/flex-server

[apfelbox]

So, a simple question: is it possible to install private packages via flex? Or is this just impossible for now?

[francoispluchino]

@apfelbox Currently, it's impossible.

[Pierstoval]

Correction: you can install private packages, because it's Composer who installs the package.
What you can't do yet is install private recipes.

[francoispluchino]

@Pierstoval Of course Composer can install private packages, the subject of the issue concerns the recipes, and so, private packages with private recipes. ;-)

[apfelbox]

@francoispluchino thank you

@Pierstoval thanks for the clarification, that is indeed what I meant. I want to install private recipes (a.k.a. private packages with these installation hooks).

Okay, that is a feature that would be really much appreciated, to use flex outside of the OSS world, too. Especially "corporate" software often times isn't OSS, but may want to benefit from these comfort features, too.

[Pierstoval]

As you can see, it's a feature a lot of people are asking for, maybe we'll get it one day :)