Document posibility of asset-packagist.org usage.

[klimov-paul]

Extracted from #8688

Posiblity of usage https://asset-packagist.org/ for asset installation istead of asset-plugin should be ducmented.

[schmunk42]

I'd also recommend to document this: https://github.com/fxpio/composer-asset-plugin/blob/master/Resources/doc/index.md#use-no-api-option-of-vcs-githhub-driver

Example: https://github.com/dmstr/docker-yii2-app/blob/21114f48843cb98282190a428b7ad8cb07401e07/composer.json#L11-L20

Background: http://www.yiiframework.com/wiki/843/boost-composer-asset-plugin-update-speed/

[cronfy]

Is there any documentation available for now?

I do not understand this: if I have several projects, how do I disable fxp/composer-asset-plugin for few (replacing it with asset-packagist.org), and leave it for other.

[samdark]

What triggers usage of asset-packagist.org is a line in composer.json. If it's not there, plugin is used.

[cronfy]

But is complser-asset-plugin still used when asset-packagist.org is enabled?

Looks like it still used. When I run composer update -vvv, I get this:

...
Reading /home/cronfy/.composer/cache/repo/https---packagist.org/provider-bower-asset$typeahead.js.json from cache
Writing /home/cronfy/.composer/cache/repo/https---bower.herokuapp.com-packages/typeahead.js-665dbadca05dcd772922b7ef32a1a6a407597cac-package.json into cache

!!! composer-asset-plugin starts
Adding VCS repository bower-asset/typeahead.js
Downloading https://api.github.com/repos/twitter/typeahead.js
Downloading https://api.github.com/repos/twitter/typeahead.js/contents/bower.json?ref=master
Downloading https://api.github.com/repos/twitter/typeahead.js/commits/master
Downloading https://api.github.com/repos/twitter/typeahead.js/tags?per_page=100
Downloading https://api.github.com/repos/twitter/typeahead.js/git/refs/heads?per_page=100
!!! composer-asset-plugin ends

Reading /home/cronfy/.composer/cache/repo/https---packagist.org/provider-yiisoft$yii2-gii.json from cache
Reading bower.json of bower-asset/typeahead.js (v0.11.1)
Reading /home/cronfy/.composer/cache/repo/github.com/twitter/typeahead.js/bower-588440f66559714280628a4f9799f0c4eb880a4a from cache
Importing tag v0.11.1 (0.11.1.0)

Assets are installed into vendor/bower-asset, thus they are installed by asset-packagist, and not by composer-asset-plugin. But composer-asset-plugin still trying to resolve these requirements, slowing down the update.

[samdark]

Is typehead dependency registered at asset-packagist.org? If not, you need to manually register it. Will be fixed one day but that's how it is now.

[cronfy]

Is typehead dependency registered at asset-packagist.org?

Yes it is: https://asset-packagist.org/site/search?query=bower-asset%2Ftypeahead.js

This is not the case, same issue with jquery:

Reading /home/cronfy/.composer/cache/repo/https---asset-packagist.org/provider-bower-asset$jquery.json from cache
Reading /home/cronfy/.composer/cache/repo/https---packagist.org/provider-bower-asset$jquery.json from cache
Writing /home/cronfy/.composer/cache/repo/https---bower.herokuapp.com-packages/jquery-6f2ab963b5a51d155d69ac091e52e506d1055057-package.json into cache
Adding VCS repository bower-asset/jquery
Downloading https://api.github.com/repos/jquery/jquery-dist
Downloading https://api.github.com/repos/jquery/jquery-dist/contents/bower.json?ref=master
Downloading https://api.github.com/repos/jquery/jquery-dist/commits/master
Downloading https://api.github.com/repos/jquery/jquery-dist/tags?per_page=100
Downloading https://api.github.com/repositories/28825109/tags?per_page=100&access_token=c9955231d071dc8afd33b18d14597cf9e2fcdd58&page=2
Downloading https://api.github.com/repos/jquery/jquery-dist/git/refs/heads?per_page=100

[samdark]

Then it's better to ask @hiqsol and @SilverFire.

[schmunk42]

@cronfy If you have performance issues, you might give this config for the asset-plugin a try:

    "extra": {
        "asset-installer-paths": {
            "npm-asset-library": "vendor/npm",
            "bower-asset-library": "vendor/bower"
        },
        "asset-vcs-driver-options": {
            "github-no-api": true
        },
        "asset-pattern-skip-version": "(-build|-patch)"
    }

It will not query the GitHub API, but use local git repos, which is several times faster after the first run. Make also sure you've a GitHub token.

Would be nice if you could give some feedback, since there are some other performance improvements possible, but we'd need to know if the current setup works as expected.

[cronfy]

@schmunk42 Here are some stats for composer update of my project with your config compared to others.

All assets were installed correctly in all cases.

1. First run was VERY slow, I did not measure, but I feel it took more than 10 minutes to clone all these repos.
2. When repos are in cache with your config: 29-36s.
3. Without your config: 1m3s-1m20s.
4. With asset-packagist.org (no composer-asset-plugin): 8-18s.

But my project uses private repo that can take 10±5 sec to update, so measures may be not very accurate.

I also tried composer.json with only one require - yiisoft/yii2.

1. First run - did not test.
2. When repos are in cache with your config: 12-17s.
3. Without your config: 35-40s
4. With asset-packagist.org (no composer-asset-plugin): 2-6s.

So, asset-packagist is more preferable to me. I just would like to know how would I use composer-asset-plugin for some projects and asset-packagist.org for others, because I can't switch to asset-packagist.org for all projects at once.

The problem is that I do not understand how to disable composer-asset-plugin on per-project basis.

[hiqsol]

@cronfy why can't you use asset-packagist for all projects? We use it for really complex setups with lots of assets.

composer-asset-plugin is installed globally (more precisely per user basis) and AFAIK there is no way to disable certain plugin on per project basis.

It is possible to disable plugins at all, but it will disable yii2-composer too and you'll get non working setup.

What you can do is use alternative composer home like /home/user/.alt-composer
Put there composer.json with (or without, which you prefer) composer-asset-plugin and then run it with command (better create script):

(export COMPOSER_HOME=/home/user/.alt-composer; composer update)

This way you can control which composer to use with or without composer-asset-plugin.

[schmunk42]

@cronfy Thank you for the feedback, it's about that what can be expected at the moment.

First run was VERY slow, I did not measure, but I feel it took more than 10 minutes to clone all these repos.

Yes, you have to take that penalty once per machine.

How often do you run composer update vs. composer install during development? I am asking because still many people run update all the time, which is only necessary if there's really an update or a new package required.

Do you commit the composer.lock file in your projects?

[samdark]

@schmunk42 why would you run composer install on dev machine at all?

[cronfy]

@samdark @schmunk42 I run composer update when I push to repos that are dependencies for the project. I do it quite often. And I run composer install when dependent repos were updated by someone else in the team. I think it's a common workflow when you develop a project and it's dependencies together.

@schmunk42 Yes, I commit composer.lock. It is harder to measure/test composer install execution time for different configs, because it would require a new commit to dependent repo for each test. When everything is up to date, composer install runs fast.

[cronfy]

why can't you use asset-packagist for all projects? We use it for really complex setups with lots of assets.

@hiqsol I can use it for all projects, but I can't switch to it for all projects in a short time. Until I switch to it in all projects, I can't remove composer-asset-plugin, and until I remove composer-asset-plugin, there is no benefit from switching.

Thanks for suggesting alternate composer for switched projects. I was hoping I could avoid it, but if there is no other way to disable composer-asset-plugin for a single project, I will use this solution.

[cronfy]

If there was a composer-asset-plugin fork, that disables itself when asset-packagist.org is enabled (or by some key in the "extra" config), it would be a better solution.

[rob006]

why would you run composer install on dev machine at all?

Because yii2 docs says that? :D

[samdark]

@rob006 right, that's when you want the same environment as others after pulling code.

[schmunk42]

@schmunk42 why would you run composer install on dev machine at all?

Because it's much much faster and does not require any plugins to be installed at all, since all information is present in composer.lock.

In our workflows there's usually only one or two developers, which run composer update actively -
every one can do this, but it's only need for new extensions or if you really want an update. Since we're trying to work with stable tags whenever possible, there are not much updates anyway. But this workflow applies even more if you have a lot of dev-master versions in your project.

I have written this on our whiteboard in the office

composer update may fix and break things - composer install just works*

(*) assuming that the tests are green ;)

So after composer update the "master-developer" commits the lock file and developers more or less run composer install "all-the-time" (best would be a git-hook, which does this after composer.lock has been changed, but anyway) - because this action takes only around 1 second.

See this gist for an example: Run composer install two times on a project which actually requires "assets", but with a vanilla composer installation (used a vanilla composer binary via Docker, no caches, no plugins, even a host-volume for the vendor folder).

[schmunk42]

I run composer update when I push to repos that are dependencies for the project. I do it quite often.

@cronfy You mean like you want to work with repos for some of your extensions?

[samdark]

@schmunk42 means that update performance is as important as install performance...

[schmunk42]

@rob006 That's the workflow for Yii contributors, not extensions per se. Or did i get that wrong?

@samdark You should include composer.lock at least in releases, my opinion is to include it always, simple because less things break. Then install should be the workflow. And there should also be no asset-plugin errors (see gist example), when with a missing plugin.

Btw: Is all the npm stuff required? Wouldn't a strict composer workflow be preferable? Like putting the npm packages into devDependencies: {"npm-asset/abc"}. Notice: I haven't tried the JavaScript tests yet.

means that update performance is as important as install performance...

Let's put it like this, install has no issues. For sure it would be nice if update would be faster.
[edit]
But I'd say that this "should" have like a 10:1 ratio.

That's also why I asked if @cronfy wants to work with repos, since he said

I run composer update when I push to repos that are dependencies for the project

Why run composer update when you want to push?! Composer can't push for itself.

---

I don't want to impose a certain workflow to developers, but you can compare both actions like so: Imagine a Debian Distribution, running composer install corresponds to apt-get upgrade, while a composer.lock` (required) would correspond to a package.gz from apt-sources.

Only (Debian) maintainers run composer update to create a new composer.lock (package.gz).

PS: git pull of your project would correspond to apt-get update.

---

PS2: Using composer update all the time is like working on Gentoo 🙊 😸

[rob006]

That's the workflow for Yii contributors, not extensions per se. Or did i get that wrong?

My point is: if you work with other people on the same project, you probably will run composer install more often than composer update, so I give first example when you want to run composer install on dev machine. And this applies not only to yii2 repos - most of the teams should use composer install in their workflow just to avoid merge conflicts or unrelated bugs caused by updated dependency .

[schmunk42]

@rob006 I am sorry, I didn't get the sarcasm in the first comment - shame on me LOL; OMG.

[cronfy]

I run composer update when I push to repos that are dependencies for the project. I do it quite often.
@cronfy You mean like you want to work with repos for some of your extensions?
Why run composer update when you want to push?! Composer can't push for itself.

@schmunk42 I have a project with composer.json that requires a module. Module is also used for other projects. I develop both project and module. Oftenly adding new functionality to the project means also adding new functionality to the module. I also do many refactoring actions when I move project's code to the module.

When work is done, I need to do this:

1. commit/push module
2. composer update <---- here it is
3. commit/push project

[schmunk42]

@cronfy I share your pain, because we have the exact same workflow. And there are use-cases where you need to run update that's out of question. We did practically only update for a long time, since you usually get repos, but that's only a side-effect.

What you should do by all means, is setting preferred-install.

Defaults to auto and can be any of source, dist or auto. This option allows you to set the install method Composer will prefer to use. Can optionally be a hash of patterns for more granular install preferences.

See here for an example.

    "config": {
        "preferred-install": {
            "dmstr/*": "source",
            "hrzg/*": "source",
            "schmunk42/*": "source",
            "*": "auto"
        }
    }

This is how a vanilla install should look like then; cloning only repos from dmstr, hrzg and schmunk42.

Bonus: Some info about the composer update workflow. (I've just tweaked this :))

[mikehaertl]

@hiqsol I've tried asset-packagist and it looks really great. Only problem I have: bower/npm assets are still installed to default location (vendor/bower-asset instead of vendor/bower) and Yii2 doesn't like this. Is there an easy fix for this?

I already tried with asset-installer-paths in the extra section of my composer.json but that's probably only for the fxp asset plugin.

[hiqsol]

You can use appropriate aliases:

    'aliases' => [
	'@bower' => '@vendor/bower-asset',
	'@npm'   => '@vendor/npm-asset',
    ],

This question is completely covered at the about page.

[mikehaertl]

Great, thanks, I missed that, as I just followed the recommendation in one of your other comments.

Coming back to the initial issue: If asset-packagist is added to the docs here, this should of course also be added there.

[schmunk42]

@hiqsol I remember from somewhere, that it is not recommended to use asset-packagist and fxp.

What was reason the behind this? Because composer does not prefer asset-packagist, because it's a "composer" repo. And therefore all asset-packages are getting scanned anyway?

@francoispluchino Would it be possible to detect if a repo would be available via asset-packagist and skip it?

Moreover, to both of you, wouldn't it make sense, that the asset plugin sends an API request to asset-packagist in such an setup? So we could have the best of all worlds, fast install + fallback + auto-registration of new asset packages for the next run.

[francoispluchino]

1. It's not possible but you can disable the plugin since the 1.3.0 version.
2. I have already made this proposal but they didn't like the installation in global mode.

[schmunk42]

I have already made this proposal but they didn't like the installation in global mode.

I mean just sending API requests like: https://asset-packagist.org/site/search?query=%PACKAGENAME% so packages in use get scanned.
What does it has to do with global mode?!

[francoispluchino]

This plugin. You can use they server without my plugin. and it's an argument for them. They will always continue to purpose their solution based on the plugin, and they are not interested to merge the 2 solutions.

As far as me on, I am open to any integration to improve the performance.

[schmunk42]

@hiqsol Would it be OK for you if the plugin would trigger an API request, when an additional option is enabled?

It's not about merging the two solutions, only to make it easier to switch to asset-packagist.

As an example, if you've updated a project with the API search triggers, you can install the project from now on from asset-packagist (as long as you don't add more - unknown - asset-packages)

[SilverFire]

Sure, we can add API to import new asset packages that are in use

[SilverFire]

Resolved by #14025