-
-
Notifications
You must be signed in to change notification settings - Fork 156
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add an option to disable asset git/GitHub API requests #251
Comments
Add command option is not possible, so only env variable is possible. |
Ah sorry, I mixed that up with new commands, but I think it does not make sense (or maybe it's not even possible) to have something like |
This, it's possible. |
So I looked a bit into this and if you skip updating the remote git server the performance gain are about ~30-50% (!). I.e. commenting this line. Updates still worked if the cache is filled. Another huge improvement would be to load
Looks to me like the file is always downloaded and then written to the cache. Maybe this could also require some changes in composer itself. Do you think it's worth to create an issue like "Feature: Allow updates from cache only"? Last but not least it would be awesome if Git updates could be run in parallel, like downloads with prestissimo. I think this would make the above changes obsolete anyway. |
This could be combined with a functionality checking an expire setting, like: "Warning Asset package definitions have not been updated for 7 days".
While I'd also like a separate command, the ENV variable is the way to set a default. @francoispluchino Any advice where to start or do you have plans about this feature, it would be great improvement. related: yiisoft/yii2#13064 |
The bower.json and package.json files are cached. It only has files in the main branch that are not cached. Regarding the You can make a pull request for this feature! |
I like the syntax |
So, @handcode and I dug a bit on this ... To really get a huge performance gain, we'd need to address two things. One is this https://github.com/fxpio/composer-asset-plugin/blob/master/Repository/BowerRepository.php#L58 We hacked it like so:
This skips the update of the git repository, if it exists, actually by setting the URL to a local path. The other one is the URL of the repo itself, see https://github.com/fxpio/composer-asset-plugin/blob/master/Repository/BowerRepository.php#L34
This one is harder to fake, because we don't know the hash of the repo. The files we need would be in [update] Would also be great if we could just delegate this to So, both points mentioned above would essentially skip the update of asset repositories.
Would love to hear some feedback about this. |
@schmunk42 Do not forget NPM Repository. Otherwise, your idea is interesting. Regarding the manipulation of the plugin options, I have already transferred the configuration of the plugin (formerly Furthermore, I will add support for the global configuration (#273), as well as support of the environment variables to change the configuration, to override the configuration with formatting: In this way you will can propose your solution to disable the analysis of the asset dependencies. |
I haven't tested this, but it looked to me, like this would also be the way npm works with fxp, simply because composer works that way. |
Your last comment change the behavior directly on the |
So, I played around a bit with npm-assets ... One thing I noticed is, that despite the fact npm is handing out a lot more information than bower, it takes only about 0,2 secs to get that information compared to 0,6 secs for bower (which is only the repo info) - I am speaking of The rest of the processes are the same, pruning and updating the repo takes the most amount of time, usually ~1-3 secs per asset repository (depending on the size)
This could also be almost completely eliminated when skipping that check by using a local repo. Another option would be to optimize that step, but that would also involve composer itself. |
Some more findings ... The line in composer which runs the update is here. Taken from https://git-scm.com/docs/git-remote - just configure that origin has not to be updated - or do that manually. I know that this somehow defeats the purpose of "update" but on the other hand, you do not have to update & purge your local mirror every time - other package manager warn you, if this hasn't been done for some time (eg. two weeks on Macports). I hardy dare to CC: @Seldaek - but I'd really like to hear his opinion about it. |
@schmunk42 We can avoid making the 5-6 queries by using the cache and checking the update date on the first query (only for the Github driver). Currently, Each repository addition systematically performs 5 minimum requests (More if several pages of tags and/or branch). Example for Jquery with Bower:
The result of:
contains the With this cache, each repository addition systematically performs 1 minimum request, and not 5. Of course, this optimization can come in addition to your proposal. |
All my tests already have the IMHO a decent performance can only be achieved, when disabling the API requests at all. And reading the tags from local git clones. I think bower, npm, yarn, ... do the same, or? (not 100% sure) |
Does it impact |
As already mentioned in the issue #226, Bower use a Bower can use a 'slow clone' (full clone) or 'fast clone' (shallow clone), see the code here. The 'slow clone' is used if the resolution use a commit id (sha). Bower verifies whether the server supports shallow cloning. This is done according to the rules found in the following links:
Summary of the rules: (extracted from the source)
The above should cover most cases, including BitBucket. IMHO, this optimization should be directly integrated to Composer, because it will benefits all VCS Repositories. |
That could be a huge improvement in install times for PHP packages as well. |
@schmunk42 Your proposal can be available for the |
I played around a bit with shallow clones, but the operation mentioned in #251 (comment) doesn't seem faster on a shallow clone. For sure the initial download is faster, but you usually have to take this penalty only once. I can't really dig it up right now, but I remember a discussion on CocoaPods that shallow clones might be even slower under some circumstances, anyhow ... I don't think they will bring much improvement overall. What costs so much time is that fxp creates vcs repositories, which are fully scanned on every update by composer. If you have 20 asset repos you need about 10 seconds for downloading the repo info (bower) and ~30 secs for updating the repo (syncMirror) - the rest of the update, like downloading info from packagist and SAT solver take usually less than 10 seconds. The former two tasks are usually not needed on every update, maybe once a day would be totally fine (or on demand). The (cache) info would be shared across projects on your host. But when you need to run update, let's say 5 times, because you need to find a matching set of extensions, this becomes really annoying - even more with API requests. |
I was re-examining this problem, and trying to remember why the discussion had stopped on this subject. The shallow clone may become slower, if for example, you have to download each branch and tag to retrieve the We cannot do this with Composer because the solver SAT retrieves the full list of dependencies with all versions even if the version of the package doesn't match. Therefore the lazy loading is not used. And if we implement the clone shallow, to retrieve the So this option is not possible without a modification of the solver SAT (and they do not want to touch it). To return to your proposal to "disable" the search of update of assets, which in the case that you explosed, and actually interesting, do you have a PR? |
Not yet :) One thing would be to skip syncMirror in composer - could this be solved in a plugin? |
Added by #289 |
Continued from yiisoft/yii2#8688 (comment)
Proposal: Add an option, like
--skip-asset-info-update
or ENV variableFXP_SKIP_ASSET_INFO_UPDATE
to disable git and/or GitHub API requests, which query for new versions/tags, since this might take a while.The option should only have an effect, when local information about the repository already exists. If not the plugin should download the information.
The text was updated successfully, but these errors were encountered: