Common files dispatcher for Symfony CMF repositories

[damienflament]

After some refactoring of Travis CI configuration files on some Symfony CMF repositories, I wanted to apply the same improvements to others repositories.

But it appears that there are a lot of differences between them. It is understandable as there isn't any centralized location (or maybe this file in the testing repository, but I'm not sure about it's usage).

@dbu pointed out the sonata-project/dev-kit project to me.

The dev-kit project

After studying the code, I think I can explain its usage.

There is 2 main utilities:

1. A web application build on Silex (see the web/ directory) which is called via GitHub WebHooks (configured automatically on the repositories) and sets up the labels applied to the issues/PRs. This application is served via a web server within a Docker container (see the docker/ directory).
2. A command line application which helps the developer to do some repetitive tasks (see the dev-kit file and the src/ directory):

Available commands:
  auto-merge               Merges branches of repositories if there is no conflict.
  depends                  Show internal sonata dependencies of each project.
  dispatch                 Dispatches configuration and documentation files for all sonata projects.
  help                     Displays help for a command
  list                     Lists commands
  merge-conflicts          Comments non-mergeable pull requests, asking the author to solve conflicts.
  pull-request-auto-merge  Merge RTM pull requests. Only active for StyleCI/SonataCI PR.

The interesting command is dispatch.

The dispatch command

This command apply common modifications to the configured repositories (see the projects.yml file) :

1. configure the issues labels (name and color),
2. configure the WebHooks,
3. dispatch the common files (see the project/ directory) after rendering them (using Twig).

The common files dispatching

The dispatch process is quite simple:

foreach configured $repository {
    clone the $repository

    foreach configured $branch {
        checkout the $branch
        create a $new_branch
        render the $files

        if the $files have been modified
           and there is not any existing PR from dev-kit
        {
            commit and push the $files to the $new_branch
            create a pull request from the $new_branch to the $branch
        }
    }
}

When the dev-kit repository is built on Travis CI, the dispatch command is run.

My proposal

I cannot take the time for implementing all dev-kit features (and I'm not sure about the usefulness of all this features for the maintainers). Even by forking it. Indeed, this project does not seem to have been designed to be shared and easily configurable. Moreover, a lot of things need to be removed to just keep what we need.

I think about implementing a simple dispatch process by using the same algorithm, using those tools:

• knplabs/github-api to create the pull request,
• twig/twig to generate the files dynamically,
• Composer to read information from the composer.json file.

Features

Some questions need an answer:

1. Is there a need to dispatch the files on multiples branches ? Or doing it on master is sufficient ?
2. Is there a need for a command line application ? Or a simple script is sufficient ?
3. Is it better to centralize the Travis configuration on the dispatcher repository (as dev-kit dot it) ? Or to host a base configuration on it and allow each repositories to override it ?

My opinion

1. The branches

I don't know how you manage this project. If you are maintaining the legacy versions, this can be useful. Otherwise, affecting only the master branch can be sufficient.

2. Command line application

A command line application is easy to implement using symfony/console. It will allow adding more tools for maintainers later and build a real development kit if needed.

3. No configuration at all

I think it's possible to generate a Travis configuration file using the Composer configuration:

• the build matrix will be generated regarding the PHP and Symfony supported versions,
• we can use two templates: the first one for the components and the second for the bundles (relying on the repository name suffix -bundle to apply the correct one).

The only configuration file will be a list of enabled repositories.

[dbu]

thanks for looking into this! sounds like a great idea!

1. branches

we said we focus on the master branches to concentrate our efforts. i think that is good enough. we can still backport stuff manually if really needed, or expand the tool later on.

2. symfony/console

implementing symfony/console is quite simple and allows things like arguments to help while developping. we could e.g. add a dry-run flag to do everything but not really creates pull requests, or to only update one repository... so i opt to use the symfony console.

3. configuration

lets start with the simplest solution and see how well that works and if we need more.

there is one thing that could be tricky: https://github.com/symfony-cmf/routing-bundle/blob/201ca313b599bda335cea350256045da05972412/.travis.yml#L42 - though maybe its good enough to run that in each repo that has phpcr-odm somewhere in the dependencies...

4. covered files

we have a bunch of files that are (hopefully) identical in all repos:

• .styleci.yml
• CONTRIBUTING.md
• src/Resources/meta/LICENSE

the readme follows a very similar pattern but has a paragraph with the purpose of the bundle. i find that paragraph very useful. while it would be cool to generate the readme consistently, we would need something to keep defining that paragraph.

if you get the travis thing running, thats a good start. once the infrastructure is there, we can add static files and maybe think of a solution for the readme.

[damienflament]

@dbu

3. configuration

lets start with the simplest solution and see how well that works and if we need more.

The simplest solution might be to use a centralized configuration file (like the dev-kit one).

But doing that way will require to modify this file on the dispatcher repository when the dependencies versions are modified in the project repository.

[dbu]

i love your idea of determining php version and symfony versions (for bundles) based on composer.json. that should hopefully not be all that complicated and would be easier to maintain. and ease of maintenance is the primary goal of dev-kit. we do need a configuration file to at least tell which repos to handle, and probably in the future exclude or override some things for some of the repo files.

[damienflament]

After some experimenting, I made a Development Kit console application with a dispatch command.
Check the damienflament/dev-kit repository and feel free clone it to experiment !

[dbu]

cool!

@ElectricMaxxx is looking into the monorepo topic. i guess that would change the needs for file dispatching as all would be in the same repository. wdyt max?
in my opinion, having a dev-kit to maintain files across all our repositories would be enough. to test upcoming features, we can also change the composer.json temporarily to use feature branches of dependencies or a specific commit id.

[damienflament]

What about the monorepo ? I usually think it's not a good practice. But I don't know why and I don't have any argument against it. I just see so many projects using multiple repos for their components. The monolithic repo is usually a meta one to allow getting the whole project.

[damienflament]

Is there a way to have a monolithic repository while keeping separate component repositories using Git submodules ?

[ElectricMaxxx]

For the monorepo:
yes there is a way. A way to to have all code for working in one directory and each repository to read. It would also be possible to make PRs on the single repositories and redirect them into the correct subtree of the monorepo, but this would be more work on syncing. Ther is an issue, where i started the discussion and started a kind of a POC in a copy of this organisation.
But we still would need some of organising Makefile, travis files and so on for each module in the mono-repo, as would also live standalone in a read-only repo.
I also forked the dev-kit into this organisation and do some experiements atm. Cause my main issue for the mono-repo is the work to maintain the packages. We are just a few devs to maintain this big amount of code, so each idea to simplify and automate the work on the repository would help, so have some kind of dev-kit (helper app for the common work) would be as nice as having the repositories in the monorepo.

[ElectricMaxxx]

the dev-kit works fine now.