Composer project importer

[kaloyan-raev]

What does this PR do?

Composer is a popular PHP dependency manager (similar to Bower for
Javascript). A natural way to create a new project in PHP is by materializing it from a Composer package like this:

composer create-project laravel/laravel /projects/laravel

The above command fetches the laravel/laravel Composer package from a central repository (https://packagist.org/) and installs it at the /project/laravel folder on the local file system. The advantage against cloning a git repository is that composer create-project will install all required
dependencies.

This PR contributes a new Composer plugin that:

• Defines a Composer project importer that can import the project source from a Composer package.
• Defines an Import Project Registrar that registers the Composer importer into the import project wizard.

The Composer project importer uses the 'location' property of the
SourceStorage model for the package name. As result the "source" of
project templates can be defined like this:

"source": {
  "type": "composer",
  "location": "zendframework/zend-expressive-skeleton",
  "parameters": {}
}

The new project importer requires the composer executable to be installed in the stack and available in the PATH. This is the usual case for all PHP stacks.

Additional changes:

• The 'Archive' importer category is renamed to 'Package'. See this comment.
• New project samples using the 'composer' source type:
  • Zend Framework Skeleton Application
  • Zend Expressive Skeleton Application

What issues does this PR fix or reference?

• PHP support PHP support #2590

New behavior

(Explain the PR as it should appear in the release notes)

• New "composer" source type is available for Project Templates. The "location" property provides the package name of the composer package. The Templates must be updated
• New "composer" project importer is available for the Import Project wizard. It allows users to create new PHP projects from the specified Composer package.
• The existing "astro-splash" project sample is updated to use the new "composer" source type.
• New project samples:
  • Zend Framework Skeleton Application
  • Zend Expressive Skeleton Application

Signed-off-by: Kaloyan Raev kaloyan.r@zend.com

[codenvy-ci]

Can one of the admins verify this patch?

[TylerJewell]

At this rate of improements, Kaloyan is going to be a Che committer before CheConf. @bmicklea @vparfonov - please schedule code review / QA into the current sprint.

[kaloyan-raev]

The second commit renames the 'Archive' importer category to 'Package'. We used to have only the Zip importer in the Archive category, but this PR also adds the Composer one. Renaming the category to 'Package' fits both Zip and Composer importers, whereas 'Archive' is not good for the Composer importer.

I kept the ARCHIVE enum class as deprecated for backward compatibility.

This commit can be easily reverted if this change is not accepted.

[bmicklea]

Very cool @kaloyan-raev!

[bmicklea]

@kaloyan-raev - is this PR ready for review? Or are you still working on this?

[kaloyan-raev]

@bmicklea The PR is ready for review.

[bmicklea]

@vparfonov - please review for M7

[gazarenkov]

Hi @kaloyan-raev

Good things and good case, congrats
But it seems importer is not really an abstraction to use for this purpose since Composer bound to PHP (unlike neutral source code storages) and it seems it has more explicit behavioural nature than just source import.

It seems PHP - Composer relationship looks exactly like Java - Maven one.
The way I propose will also allow you to use all the other functionality offered by Composer, not only project bootstrapping with "create_project" option.
So, I propose to organize it this way:

• Make Composer PT which extends PHP PT,
  see Maven PT .
  Include it into "PHP" group so in the Project Wizard it will be listed along with "bare PHP" PT see Maven wizard.
• To make it possible to bootstrap project make default Composer generator use your code with "create_project" option. Or you can make it even more advanced (maybe later) making 2 different generators (from scratch and bootstrapping, see how it is done on Maven PT for Simple/Archetype generators, see Maven generator ).

[kaloyan-raev]

@gazarenkov Thanks for the review and the advice.

Indeed, the PHP - Composer relationship is similar to the Java - Maven. I will look at the Maven plugin as you have suggested and try to rework this PR.

[gazarenkov]

Cool, feel free to ask me or @vparfonov if you need some help with that, good luck.

[kaloyan-raev]

@gazarenkov I looked at the Maven plugin. I agree that from a design point of view it would be better to move the creation of a Composer project from the Import Project wizard to the Create Project wizard.

However, the main motivation for this PR was to make it possible to have project templates that are created directly from a composer package. I would still need the project importer to achieve this, right?

[kaloyan-raev]

@gazarenkov @vparfonov Could you look at my last question?

[vparfonov]

@kaloyan-raev Hi, I don't know how composer work but look like we don't need it as importer. Importer means that you have some remote, for example VCS. If composer just generate some files like maven archetype plugin it's org.eclipse.che.api.project.server.handlers.CreateProjectHandler in our Project API.

[kaloyan-raev]

@vparfonov Quote from the composer create-project command documentation:

You can use Composer to create new projects from an existing package. This is the equivalent of doing a git clone/svn checkout followed by a "composer install" of the vendors.

So it does fetch source code. And it also installs the required dependencies. It is the preferred way in the PHP world to start coding on a project template.

My main goal is to have Che project templates created from Composer packages. I still need the project importer to achieve this, don't I? Let me know if I can achieve it in another way.

I can still refactor the GUI part and plug into the Create Project wizard, instead of to the Import Project wizard.

[kaloyan-raev]

I removed the IDE plugin that extends the Import Project wizard from this PR, so we can focus entirely on the ability to create project templates from a Composer package, i.e. with source.type = composer.

I will prepare a new PR with IDE plugin extending that Create Project wizard after we merge this PR.

[vparfonov]

@kaloyan-raev we have fixed Maven Archetype Project generator here 9951639

i think you can do the same with composer

[kaloyan-raev]

@vparfonov Can you give me an example how I can use the Maven Archetype Project generator in a project template?

[kaloyan-raev]

Conflicts resolved.

[vparfonov]

Hi @kaloyan-raev. We added new method in Project API ProjectService.java#L211 it accepts NewProjectConfigDto.java, so you can set in Map<String, String> options all parameters that you need for project generation. Also you need to register
CreateProjectHandler for your project type and move logic from your ComposerProjectImporter.java to it.

[kaloyan-raev]

@vparfonov I am trying to rework the PR as you suggest. Now I have a Composer project type extending the PHP project type. I also have a CreateProjectHandler.

But I am failing to pass any options or attributes from the project template definition in the samples.json to the CreateProjectHandler. The onCreateProject() method is called, but both attributes and options are empty.

Here is my definition in sampled.json:

{
    "name": "zend-expressive",
    "displayName": "zend-expressive",
    "path": "/zend-expressive",
    "description": "A skeleton application using the Zend Exressive framework. Expressive is a minimalist PSR-7 middleware framework for PHP with routing, DI container, optional templating, and optional error handling capabilities.",
    "projectType": "composer",
    "mixins": [],
    "attributes": {
      "language": [
        "php"
      ],
      "package": [
        "zendframework/zend-expressive-skeleton"
      ]
    },
    "options": {
      "package": "zendframework/zend-expressive-skeleton"
    },
    "modules": [],
    "problems": [],
    "source": {},
    "commands": [
      {
        "name": "configure",
        "type": "custom",
        "commandLine": "chmod 777 ${current.project.path}/data/cache; echo \"<VirtualHost *:80>\n    DocumentRoot ${current.project.path}/public\n    SetEnv APPLICATION_ENV 'development'\n    <Directory ${current.project.path}/public>\n        DirectoryIndex index.php\n        AllowOverride All\n        Require all granted\n    </Directory>\n</VirtualHost>\" | sudo tee /etc/apache2/sites-available/000-default.conf",
        "attributes": {
          "previewUrl": ""
        }
      }
    ],
    "links": [],
    "category": "Samples",
    "tags": [
      "Zend"
    ]
  }

Note that I am specifying the "package" value in both options and attributes just for the tests. I am fine using any of them. And I am not really sure what is the difference between them.

[vparfonov]

@kaloyan-raev Can you push your changes to your branch i will check it, thx?

[kaloyan-raev]

@vparfonov Meanwhile I was able to debug the problem. I created PR #3375 with a potential fix.

[kaloyan-raev]

@vparfonov I reworked the PR. I take advantage of both the CreateProjectHandler and ProjectInitHandler. In the CreateProjectHandler.onCreateProject() method I create the project from the Composer package, but without installing the dependencies. The dependencies are installed in the ProjectInitHandler.onProjectInitialized(). This way I also cover the use case where the project is imported from a source location (Git, SVN, ZIP) and then configured as a Composer project.

The output of the executed Composer commands are displayed in a Composer console output in the IDE.

The plugin also contributes a page to the Create Project wizard where the user can specify a Composer package, which the create the project from.

All of the above works pretty nice, but I face some issues that should be addressed separately:

1. The dashboard does not report any progress as console output during the "Creating project" phase. This is due to a limitation of the API. The ProjectImporter.importSources() has a LineConsumerFactory parameter, which is not available in the CreateProjectHandler.onCreateProject() and ProjectInitHandler.onProjectInitialized() methods.
2. The dashboard waits for both onCreateProject() and onProjectInitialized() to finish before showing the Web IDE. Installing the Composer dependencies may take long minutes as they are downloaded from the Internet. It's not a nice user experience to just sit and wait in the "Creating project" status of the dashboard. It would be much better if the dashboard switches to the Web IDE after onCreateProject() finishes and executes onProjectInitialized() in the background. The user will see the progress of installing the dependencies in the console and can do other tasks in the mean time.
3. Same for the Web IDE's Create Project wizard. The wizard stays modal until both onCreateProject() and onProjectInitialized() finish. It would be better if the wizard executes onProjectInitialized() in the background.
4. The onProjectInitialized() is executed twice when a new project is created from a built-in sample with the Web IDE's Create Project wizard. The second call comes from updating the project after it is created. I am not sure if this update call is really necessary. This double call to onProjectInitialized() causes issues to some Composer projects and returns with error.
5. There is some strange flickering of the Composer console output. When a new line is added the view is scrolled to the top and then back to the bottom. It's very annoying.

[bmicklea]

@kaloyan-raev I am going to move this to 5.1 - we are trying to finalize M9 which will be the 5.0 GA release and it seems there are more changes needed before this is ready to merge.

[vparfonov]

will be merged in #3726