touch(): Utime failed: Permission denied

[abrudin]

After commit 051b7bd it is no longer possible to have a shared composer cache, as the user running composer has to own the file in order to run utime which is done by touch($this->root . $file, filemtime($this->root . $file), time());

It would be desirable to only try to run utime if the user owns the file.

[Seldaek]

So we should just do a is_writable check before doing the touch? I am not sure however why this commit breaks it and why it wasn't an issue before? Is the filemtime() call failing? Because touch() by default changes both access and modification time as far as I can tell, which we still do.

/cc @joshdifabio

[joshdifabio]

@abrudin I get the following error if I change ownership of a cached file and then run composer install such that the cached file is copied:

touch(): Utime failed: Permission denied

However, I see this error even when I revert 051b7bd. Have you already tried reverting that change yourself to see if it does indeed resolve the issue?

Regardless, it might make sense only to invoke touch if the current user has sufficient privileges, but this would obviously hide issues where the cache directory unexpectedly has the wrong ownership and cache pruning would then also fail. What do you think @Seldaek?

[Seldaek]

Yeah it strikes me as kind of impossible to share the cache dir in a sensible way if all users don't have write access to it, because then the cache can't be written to.

[abrudin]

Well, every user has write access to the cache, the problem is that utime needs ownership of the file (or root access), which everybody obviously cannot have. We reverted to 1.0.0-alpha10 (from 1.0.0-dev) and then it started working again. Previously it failed on that exact line. So the problem is not write permissions, but ownership. So I guess it is the filemtime thing that invokes utime and thus fails if it is not owned by the current user. Something like if(get_current_user() == fileowner ( $filename )) {touch with filemtime} else {touch without filemtime} could probably work.

[joshdifabio]

I'm able to reproduce this error both on 1.0.0-alpha10 and HEAD.

@abrudin I think I know why it worked for you after rolling back to alpha10: whenever Composer fails to copy a file from its cache directory it will attempt to delete that file from its cache. In the case of this error, the file with the wrong (or at least different) ownership is removed by Composer from its cache during the failed build. It is then downloaded afresh and re-created in the cache directory with the correct ownership on the following build, and so that build will succeed. Even if you build repeatedly with 051b7bd you should see successful builds. I've seen this myself when reproducing this issue.

Just to be clear, touch() (and the underlying utime operation, which is failing in your case) was already invoked prior to 051b7bd. Reproducing this myself and looking at the code, I don't believe 051b7bd has any impact on this issue.

Checking the file ownership prior to invoking touch() would resolve this issue but would also prevent cache pruning from operating as expected; i.e., recently accessed files could be removed from the cache. Arguably this is an acceptable trade-off but I think that's @Seldaek's call.

[abrudin]

@joshdifabio We have used this setup for several months, and yesterday after updating composer was the first time we ran into this issue. We can also see that building several times will eventually succeed (because of what you write above), but we have a lot of libraries so that would mean a lot of fails before we are through them all. We have tested back and forth so we are pretty sure the issue exists in the latest version but not in alpha10.

That said, it does not necessary mean that it is that commit that makes it fail, I have yet not had time to bisect the exact commit, it only seemed reasonable as that was the line that failed and it was changed recently.

I am going to try to pin down the issue more in detail and setup a test case, so that it will be clear exactly what the problem is. It could be noted that PHP:s touch implementation (https://github.com/php/php-src/blob/cf50748f2ba16cef1d2931c0d8afc717dea8eab7/ext/standard/filestat.c) behaves differently depending on the number of arguments, so I would not rule the mentioned commit out completely, but as you say it seems like the issue would have been the same before as you cannot use php:s touch without ownership (according to several comments in the docs). I'll be back with some more details.

[abrudin]

Maybe I should mention that we have this issue on Ubuntu 12.04, PHP 5.5.24.

[alcohol]

The touch operation cannot be the touch root cause of your issue, because as others have mentioned, it was already present (the only thing that was changed was the value of the modified time, not the action itself).

It is more likely something else in your environment changed. Was updating composer literally the only step you took? Everything else in your build process was 100% guaranteed left untouched?

[alcohol]

On another note, there should be some more sanity checks probably to preclude scenarios such as these from breaking the flow.

[abrudin]

@alcohol: Yes updating composer was the only change. We downgraded then it worked, we upgraded again it stopped working, downgraded it worked.

As I mentioned above the touch function works slightly different when invoking it with three arguments, depending on for example if HAVE_UTIME_NULL is defined on the system, and some paths only returns false without causing an error etc (https://github.com/php/php-src/blob/cf50748f2ba16cef1d2931c0d8afc717dea8eab7/ext/standard/filestat.c#LC701), but I agree that it doesn't look like it should have worked before the change in touch() either if you do not own the file.

I hope I will get some time later today to investigate more thoroughly which commit that actually breaks it for us.

[alcohol]

Your claim that it behaves different based on number of arguments is simply not true in this scenario.

There must be something else that modified the cache behaviour in composer between those two versions. The fact that touch() never threw an error in the older release would indicate to me that that segment of code simply never ran. Therefor conditions must differ.

Edit: to illustrate why the amount of arguments is not really relevant, this is the offending/failing part.

[abrudin]

Hello again,

now I have done a git bisect to narrow down the commit that makes it fail, and it was indeed the previously mentioned commit:

051b7bd44b52db04b4b5e54db99436b834a35c21 is the first bad commit
commit 051b7bd44b52db04b4b5e54db99436b834a35c21
Author: XYZ
Date:   Thu Apr 23 16:08:03 2015 +0100

    Touch access time only when reading from cache

I also took that commit and manually edited the cache file (removed the two last arguments for touch) just to be 100% sure.
@alcohol: I do not have enough C proficiency to claim things, so see my comments as suggestions :) For example, the operation VCWD_UTIME may succeed if (for example) newtime == NULL (couldn't find source code for that call, though), so it may still depend on the previous path in the function.

I have created a test case and I will try to describe it as good as possible:
Create a new folder and set the owner to root or any user except your own. Change permissions so that it has 2775 (setgid + group write permissions). Point COMPOSER_CACHE_DIR to that directory. Create a composer.json with some dependencies, and run composer install:

export COMPOSER_CACHE_DIR="yourcachedir"
php composer/bin/composer install --prefer-dist

Change the owner of the files created in the cache (but keep the group, to similate that another user did the composer install) and remove all the files in vendor:

sudo chown -R root:yourusername yourcachedir
rm -rf vendor/*

Run again. Before commit 051b7bd it works fine, after it fails with the utime error message.
Edited: Removed email in git bisect output.

[alcohol]

Can you verify that it actually touches it though? Does it run through the discussed block of code?

[alcohol]

This is interesting: http://linux.die.net/man/2/utime

Changing timestamps is permitted when: either the process has appropriate privileges, or the effective user ID equals the user ID of the file, or times is NULL and the process has write permission for the file.

It would seem the NULL does make a difference for the underlying implementation of touch() (I traced it to utime).

[abrudin]

@alcohol: Yes, I verified that it ran the line (and that the touch call returned true when invoked with only one parameter), so it seems your finding might be the issue.

[joshdifabio]

@alcohol @abrudin well done, guys!

FYI you can prove this quite easily:

# unix command line
$ touch delete.me
$ sudo chown another:user delete.me
$ touch delete.me # succeeds
$ touch delete.me -a # access time only: fails

<?php
touch('delete.me'); // succeeds
touch('delete.me', null, time()); // fails

I think we need to switch approaches based on the file owner.

[Seldaek]

How about if(!@touch(file... with filemtime...)) { touch(file); } ? So it works as it works now for most cases, but if you do share the cache you just get the old behavior which does the right thing anyway, it was only disturbing use cases like travis where the modification time matters.

[alcohol]

@ operator makes me cringe most of the time, to be honest.

On the other hand, adding permission and owner checks is kind of elaborate for an issue this minor.

[abrudin]

Thanks for your help! I'll monitor this issue and upgrade as soon as it is fixed, but I'm convinced you will find the most appropriate solution given all possible different scenarios. If you want me to do anything more regarding this, just let me know.

[Seldaek]

Ok I pushed a tentative fix, if you can confirm it works with the latest phar that'd be great.

[joshdifabio]

Assuming Composer's error handler respects @ then that seems like the best option to me. The alternative seems over-complicated and a bit more scary:

if (fileowner(file) === posix_geteuid()) {
    touch(file, ...)
} else {
    touch(file)
}

[abrudin]

Yes, it works just fine. Thanks again for your help and impressing speed.

[willrowe]

I was having issues with this too, even after updating to the latest version. I figured there were some broken permissions in the cache so I re-installed from scratch and it's working fine now.

[Jaspur]

@Seldaek not working. Still getting: touch(): Utime failed: Permission denied

Composer version 1.0-dev (cedbe7f1a0c85a998e3d3e707e53de0781820087) 2015-08-10 11:58:58

[alcohol]

I actually get that error too sometimes. Although very sporadically. Usually if I retry it just works. Weird behaviour (OSX) :-/

[Jaspur]

I fixed it somehow on an Ubuntu VM yesterday, but another VM started today with the same behavior. But I didn't remember what I did do to fix it.

[Seldaek]

@Jaspur do you get an exception or just that line in the output and it continues? If an exception please run with -v and paste the backtrace, just to be sure where it comes from.

[Jaspur]

@Seldaek

$ composer install -v
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Analyzed 200 packages to resolve dependencies
Analyzed 478 rules to resolve dependencies
Nothing to install or update
Generating autoload files
> post-install-cmd: php artisan optimize
Generating optimized class loader
Compiling common classes



  [ErrorException]
  touch(): Utime failed: Permission denied



Exception trace:
 () at phar:///usr/local/bin/composer/src/Composer/Installer.php:347
 Composer\Util\ErrorHandler::handle() at n/a:n/a
 touch() at phar:///usr/local/bin/composer/src/Composer/Installer.php:347
 Composer\Installer->run() at phar:///usr/local/bin/composer/src/Composer/Command/InstallCommand.php:131
 Composer\Command\InstallCommand->execute() at phar:///usr/local/bin/composer/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:257
 Symfony\Component\Console\Command\Command->run() at phar:///usr/local/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:874
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/local/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:195
 Symfony\Component\Console\Application->doRun() at phar:///usr/local/bin/composer/src/Composer/Console/Application.php:146
 Composer\Console\Application->doRun() at phar:///usr/local/bin/composer/vendor/symfony/console/Symfony/Component/Console/Application.php:126
 Symfony\Component\Console\Application->run() at phar:///usr/local/bin/composer/src/Composer/Console/Application.php:82
 Composer\Console\Application->run() at phar:///usr/local/bin/composer/bin/composer:43
 require() at /usr/local/bin/composer:25


install [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--no-plugins] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [--ignore-platform-reqs] [packages1] ... [packagesN]

[Seldaek]

So that's a touch() on the vendor-dir to indicate that the dependencies changed.. Permission denied is definitely weird, but I guess I could suppress that one as if it doesn't work it's not critical to 99% of people.

[Jaspur]

Thnx

[shivas]

@Seldaek Got this on Ubuntu

[ErrorException]
touch(): Utime failed: Permission denied

Exception trace:
() at phar:///var/www/secret/releases/20151006124723/composer.phar/src/Composer/Cache.php:149
Composer\Util\ErrorHandler::handle() at n/a:n/a
touch() at phar:///var/www/secret/releases/20151006124723/composer.phar/src/Composer/Cache.php:149
Composer\Cache->copyTo() at phar:///var/www/secret/releases/20151006124723/composer.phar/src/Composer/Downloader/FileDownloader.php:127
Composer\Downloader\FileDownloader->doDownload() at phar:///var/www/secret/releases/20151006124723/composer.phar/src/Composer/Downloader/FileDownloader.php:88
Composer\Downloader\FileDownloader->download() at phar:///var/www/secret/releases/20151006124723/composer.phar/src/Composer/Downloader/ArchiveDownloader.php:35
Composer\Downloader\ArchiveDownloader->download() at phar:///var/www/secret/releases/20151006124723/composer.phar/src/Composer/Downloader/DownloadManager.php:199
Composer\Downloader\DownloadManager->download() at phar:///var/www/secret/releases/20151006124723/composer.phar/src/Composer/Installer/LibraryInstaller.php:156

[knvpk]

For me also, same error

[alcohol]

@pavankumarkatakam are you using the latest version of Composer?

[knvpk]

yes

[protagora]

... while 'Installing zendframework/skeleton-application' (osx)

[ErrorException]                          
touch(): Utime failed: Permission denied  

Exception trace:
 () at phar:///usr/local/bin/composer/src/Composer/Cache.php:151
 Composer\Util\ErrorHandler::handle() at n/a:n/a
 touch() at phar:///usr/local/bin/composer/src/Composer/Cache.php:151
 Composer\Cache->copyTo() at phar:///usr/local/bin/composer/src/Composer/Downloader/FileDownloader.php:127
 Composer\Downloader\FileDownloader->doDownload() at phar:///usr/local/bin/composer/src/Composer/Downloader/FileDownloader.php:88
 Composer\Downloader\FileDownloader->download() at phar:///usr/local/bin/composer/src/Composer/Downloader/ArchiveDownloader.php:35
 Composer\Downloader\ArchiveDownloader->download() at phar:///usr/local/bin/composer/src/Composer/Downloader/DownloadManager.php:199
 Composer\Downloader\DownloadManager->download() at phar:///usr/local/bin/composer/src/Composer/Installer/LibraryInstaller.php:175
 Composer\Installer\LibraryInstaller->installCode() at phar:///usr/local/bin/composer/src/Composer/Installer/LibraryInstaller.php:89
 Composer\Installer\LibraryInstaller->install() at phar:///usr/local/bin/composer/src/Composer/Installer/InstallationManager.php:152
 Composer\Installer\InstallationManager->install() at phar:///usr/local/bin/composer/src/Composer/Installer/InstallationManager.php:139
 Composer\Installer\InstallationManager->execute() at phar:///usr/local/bin/composer/src/Composer/Installer.php:604
 Composer\Installer->doInstall() at phar:///usr/local/bin/composer/src/Composer/Installer.php:232
 Composer\Installer->run() at phar:///usr/local/bin/composer/src/Composer/Command/CreateProjectCommand.php:172
 Composer\Command\CreateProjectCommand->installProject() at phar:///usr/local/bin/composer/src/Composer/Command/CreateProjectCommand.php:130
 Composer\Command\CreateProjectCommand->execute() at phar:///usr/local/bin/composer/vendor/symfony/console/Command/Command.php:256
 Symfony\Component\Console\Command\Command->run() at phar:///usr/local/bin/composer/vendor/symfony/console/Application.php:838
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/local/bin/composer/vendor/symfony/console/Application.php:189
 Symfony\Component\Console\Application->doRun() at phar:///usr/local/bin/composer/src/Composer/Console/Application.php:167
 Composer\Console\Application->doRun() at phar:///usr/local/bin/composer/vendor/symfony/console/Application.php:120
 Symfony\Component\Console\Application->run() at phar:///usr/local/bin/composer/src/Composer/Console/Application.php:98
 Composer\Console\Application->run() at phar:///usr/local/bin/composer/bin/composer:43
 require() at /usr/local/bin/composer:25

[tomsihap]

It seems that composer clearcache solves the issue.

[shivas]

composer clearcache is not an option in some jenkins or other CI tools. Clearing cache and re-downloading all those packages again increase build time.

[curry684]

It's quite probably a cache corruption. As far as I can see in the code however there is no real reason for it to be fatal - worst that could happen is that it gets cleaned up too soon. Pushing a PR in a sec.

[kala725]

try running sudo composer clear-cache and then run the composer install , because touch is required when it reads from cache & when cache is cleared, that is not executed at all.

[ooples]

I tried composer clear-cache and that didn't fix the issue for me. Please help!

[Vitaliy070516]

Thank you very much, kala725. sudo composer clear-cache helped me at once ! )))

[samokspv]

Same error, but change permissions, 'sudo composer clear-cache' or remove vendor dir and ''composer update' isn't help.
$ tail laravel-2018-06-06.log
App\Http\Controllers\Controller::render(21): Exception
exception=touch(): Utime failed: Permission denied

Laravel v5.5.40
Debian GNU/Linux 9.4
Composer version 1.6.5 2018-05-04 11:44:59

[alcohol]

That's not a Composer error.