Packages don't update to latest commit / Package repo fails to update

[rdohms]

When declaring packages inside the compose.json file, if reference points to master then running updates does not check out the latest commit in the master branch.

Expected behaviour:
Composer should see that branch is master and check if commit hash is the same as HEAD, if not it should checkout head.

Actual behaviour:
Composer sees that branch is master and ignores the rest.

Temporary workaround is to add the actual commit hash into the reference field, which gets the desired outcome but add overhead in updating.

Ideally reference should also accept dev-master as code for always getting the latest commit, or simply update to latest commit in everything but tags.

[Dynom]

+1 I think I had issues related to something this feature could fix. During deployment some servers failed to checkout the correct commit object (Git)

[mevers47]

+1 I got exactly the same issue as Dynom has.

[Seldaek]

It's unfortunately not easy to fix, because it would mean we should check the git repo for changes before even running the solver, just to know if your master is up to date with the real master. Alternatively we just update packages with such references if they are already installed. So every time you'd run update, it would update this package to master, no matter if there are new commits or not. It would look awkward IMO and would not feel like the other packages.

[patcon]

I believe how Bundler/Librarian handle this is to store the commit hash in the lockfile whereas Composer is currently storing a branch reference in "source-reference": "master".

I believe this would also help deal with #448

If we store the hash, then we can do some cool stuff with git rev-parse, git cat-file, and maybe git ls-remote to ensure there's no ambiguity.

Related to storing hashes in lockfiles, you can quickly see how other package managers are doing this by searching their repos with git grep 'rev-parse'. This seems to be the common approach to ensuring lockfiles unambiguously reflect what we want. NPM was the only package manager I could find that didn't follow this approach, so I suppose we need to decide whose example we want to follow. My view is that the lockfile is always supposed to be timeless and canonical -- it should be locked. If it can't be that, then it's not living up to the promise of a lockfile, imho :)

//cc @yfeldblum @RobLoach

EDIT: I've since realized that npm doesn't actually do any dependency resolution, and instead recursively stores every version of a required module at each level. It was relayed to me that this has something to do with not really caring about loading everything into memory in JS like you would in another language. So while NPM is the only outlier, it's for good reason, and we likely want to follow the lead of other package managers that actually do dependency resolution.

[Seldaek]

There are two issues there, the lock file in one thing, and indeed we could just read the commit hash when locking instead of locking what is in source-reference (at the moment it already does that to get the commit date in case the reference becomes invalid).

The other issue is a bit trickier, and it is that of managing updates. The best way is really to add a pass between the solver and the execution of the operations that forces those packages to update if they're not being modified, but only if they have new commits. This means a git fetch every time for every package, which isn't always too fast, but I guess it's doable.

[patcon]

Thanks man.

we could just read the commit hash when locking instead of locking what is in source-reference

So you mean adding a source-revision key to composer.lock? Would the idea be that it would become to first-class citizen in terms of checking out git repos (as opposed to source-reference)? As in, you would specify a branch/tag reference, but it would be resolved into the revision in composer.lock, and this would be checked out on composer install?

This means a git fetch every time for every package

Would it not be quicker to compare git rev-parse <BRANCH> (for the local branch head commit hashes) against git ls-remote <REMOTE> <BRANCH> (for the remote branch head hash)? ls-remote is one of the few git commands that works on remote repos without fetching, and I believe it should always be pretty quick.

[penfold45]

Hi was any progress made on this? Could really do with it.

[frangeris]

👍 I got exactly the same issue, does not resolved yet? I've tried using "@dev" but don't work... :(

[sp-juan-hernandez]

Same issue.

Composer version 1.0-dev (79ac2ca) 2016-02-04 12:40:23

I even tried:

• rm -rf vendor
• cache-clear
• rm -rf ~/.composer
• selfupdate, reinstall and a previous version

[sp-juan-hernandez]

I copy the commit hash and paste it on the lock as a workaround. It works as further updates also don't change de commit.

[k1sul1]

Bringing this issue up, because this truly rustles my jimmies. I accept that Composer will never work like you'd hope it work; e.g. dev-master always checking the actual master and then checking out the latest commit.

But when you have a package at packagist.org, which is autoupdated when a commit is pushed into the package repository, and you can see the commit hash change, it's infuriating to see Composer report with "Nothing to install or update" when running composer update.

This is a massive UX problem, and causes hate against Composer because it's unintuitive when you compare it to other package managers such as npm. 5 out of 5 people that I've introduced to Composer hate it because of reasons like this. I'm not kidding. It's pretty hard to get them to adopt it after they hate it.

The problem could be easily mitigated by extending the response of composer update. Simply tell users what to actually do (add commit hash to the constraint) if they're using dev-*, instead of making them search for an answer. I've used Composer for quite a while and I still stumble upon this issue every now and then.

The other issue that I'd like to mention; Getting a minimum stability error when trying to install a package is infuriating, as it doesn't even tell you what to do. Instead you're meant to read the docs and google the error message with correct keywords to get a solution. You can rinse and repeat this for just about any error :')

[robertoandrade]

I've bumped into this a couple of times and was scratching my head as to why it worked on some of my repos and not others. I've noticed that in the repos where it works "as expected", I have the following configuration on my composer.json:

"config": {
        "preferred-install": "dist"
},

I suppose when that's the case the behavior for composer to pull the sources is slightly different (ie: it doesn't git clone the repo, since it seems to default to source as the preferred-install config).

[Seldaek]

@k1sul1 @robertoandrade if you can describe a bit more what you mean, or possibly give a way to reproduce this, I'd be curious. As far as I know updates to a dev-master version work perfectly fine. Is the problem when you did a git checkout of some other branch inside the vendor dir? Or doing a simple update is not updating?

[k1sul1]

If I have dev-master spesified as the version on a subpackage I'm working on, I expect composer update to always fetch the latest commit. That has not been the case. IIRC I had to remove the entire lockfile and the vendor directory to get the latest update.

Ofc, I could've used composer remove vendor/package && composer require vendor/package:dev-master, but that literally took multiple times more than simply nuking them out of the orbit.

[Seldaek]

@k1sul1 ok, that sounds very odd.. How is your "subpackage" loaded, is it on packagist? path repo? vcs repo? If you can make a repro case of this somehow I'd be curious to see, because as far as I can tell it works.

[k1sul1]

It was on packagist, yeah. I tried using VCS once, stopped using it immediately because it was too slow on updates. I wanted symlink-like behaviour instead of waiting minutes. Packagist gave me that. I used to develop an actual project in a virtual machine, and the dependency of the project in another VM, so actual symlinking wouldn't have been possible.

When you say "as far as I can tell it works", do you mean it works with 2.0, or should it work with 1.x too?

Only PHP development I do these days is WordPress, and before the set of plugins required to use WordPress with Composer is updated for 2.0, I can't test & verify it myself.

Here's one of my packages: https://github.com/libreform/libreform

If you were to require it with composer require libreform/libreform:dev-master, composer update wouldn't update the package after I push an untagged commit. Only way I was able to get precise commits, was to use the commit hash directly. If that's not a problem anymore, great!

[Seldaek]

Ok yeah I have a hard time believing this sorry :D I am 100% sure that works for me, already in v1. It might take a minute or two for the metadata update to happen (that is a bit faster for composer 2), but otherwise it should absolutely work.

[sypets]

@rdohms can you check if still reproducible?

It looks like Composer will now update package correctly if required as dev-master (via Packagist).
With composer version 2.1.9 and a package required as "dev-master" I get:

• composer outdated shows me the new commit hash if there is a newer one
• composer update vendor/package updates the package

e.g.

---

1. in composer.json:

"vendor/mypackage": "dev-master",

---

2. composer outdated:
   Result:

vendor/mypackage                      dev-master 350a877 ! dev-master e00b00d ...

(shows newer commit hash)

---

3. composer update vendor/mypackage:
   Result:

 - Upgrading vendor/mypackage (dev-master 350a877 => dev-master e00b00d)
...

---

4. composer.lock:
   diff

-                "reference": "350a877949d81008a09c08aa928235451bce48c6"
+                "reference": "e00b00db5f08b70ee56a8316860300bcb7b6e70f"
etc ..

[reinholdfuereder]

@sypets I just stumbled over (something like) this as well: trying to update a custom dev-master package dependency:

• with composer v1 (1.10.6) it sees outdated dev-master package, but does not update it
• with composer v2 (2.1-dev) it not only sees outdated, but also updates it (as expected) 👍

[Seldaek]

Closing this here as I really don't see what to do about it. If someone can still reproduce this, please reach out to me before changing anything and then maybe I can inspect things on packagist.org.

[crabique]

Not sure if this is still relevant, but I am able to reproduce this issue right now using latest Composer version.

It sees 84093d0 as the latest commit for some reason, while it should be 009de30 as it's the HEAD. It works if I pass the commit hash explicitly but this is not ideal.

Also if I run it without the hash afterwards it actually downgrades it back to 84093d0 🙂

~/.wp-cli/packages # composer --version
Composer version 2.3.10 2022-07-13 15:48:23

~/.wp-cli/packages # composer require runcommand/precache:dev-master
./composer.json has been updated
Running composer update runcommand/precache
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking runcommand/precache (dev-master 84093d0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing runcommand/precache (dev-master 84093d0): Extracting archive
Generating autoload files

~/.wp-cli/packages # composer require runcommand/precache:dev-master#009de30
./composer.json has been updated
Running composer update runcommand/precache
Loading composer repositories with package information
Updating dependencies
Lock file operations: 0 installs, 1 update, 0 removals
  - Upgrading runcommand/precache (dev-master 84093d0 => dev-master 009de30)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 1 update, 0 removals
  - Downloading runcommand/precache (dev-master 009de30)
  - Upgrading runcommand/precache (dev-master 84093d0 => dev-master 009de30): Extracting archive
Generating autoload files

[wouter-toppy]

Not sure if this is still relevant, but I am able to reproduce this issue right now using latest Composer version.

It sees 84093d0 as the latest commit for some reason, while it should be 009de30 as it's the HEAD. It works if I pass the commit hash explicitly but this is not ideal.

Also if I run it without the hash afterwards it actually downgrades it back to 84093d0 slightly_smiling_face

~/.wp-cli/packages # composer --version
Composer version 2.3.10 2022-07-13 15:48:23

~/.wp-cli/packages # composer require runcommand/precache:dev-master
./composer.json has been updated
Running composer update runcommand/precache
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking runcommand/precache (dev-master 84093d0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Installing runcommand/precache (dev-master 84093d0): Extracting archive
Generating autoload files

~/.wp-cli/packages # composer require runcommand/precache:dev-master#009de30
./composer.json has been updated
Running composer update runcommand/precache
Loading composer repositories with package information
Updating dependencies
Lock file operations: 0 installs, 1 update, 0 removals
  - Upgrading runcommand/precache (dev-master 84093d0 => dev-master 009de30)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 1 update, 0 removals
  - Downloading runcommand/precache (dev-master 009de30)
  - Upgrading runcommand/precache (dev-master 84093d0 => dev-master 009de30): Extracting archive
Generating autoload files

I had the same while trying to get the latest commit of PrintNote

Same Composer version, cleared cache and all that stuff. but didnt work.

passing the commit hash worked

[xtfer]

Same issue. composer update doesnt work, but adding a commit hash does.

[Seldaek]

@xtfer if you don't provide details (package name, reference you get and reference expected at least) this is really not something we can look at..

[jrrdnx]

@Seldaek If it helps, my installed version of born05/craft-twofactorauthentication is 3.3.2, but both GitHub and Packagist show the current version as 3.3.3.

Running composer outdated shows that an update is available:

% composer outdated
Color legend:
- patch or minor release available - update recommended
- major release available - update possible

Direct dependencies required in composer.json:
born05/craft-twofactorauthentication 3.3.2     3.3.3    Craft 4 plugin for two-factor or two-step login using Time Based OTP.

But the package isn't updated after running either composer update or composer update born05/craft-twofactorauthentication. Both return:

% composer update born05/craft-twofactorauthentication
Loading composer repositories with package information
Updating dependencies                                 
Nothing to modify in lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove

[xabbuh]

@jrrdnx Please share your composer.json and composer.lock files.

[Seldaek]

3.3.3 changed two dependencies to new major versions.. So I think probably you just need to run composer update born05/craft-twofactorauthentication --with-dependencies. If that doesn't work then check composer why-not born05/craft-twofactorauthentication 3.3.3

[jrrdnx]

Thanks @Seldaek, it is due to those major version changes. --with-dependencies did not work, but why-not confirmed it.