Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why composer update break emoji on various environment? #8268

Closed
UksusoFF opened this issue Aug 8, 2019 · 16 comments

Comments

@UksusoFF
Copy link

commented Aug 8, 2019

My composer.json:

{
    "name": "REMOVED",
    "description": "REMOVED",
    "keywords": [],
    "license": "MIT",
    "type": "project",
    "require": {
        "php": "7.2.*",
        "ext-PDO": "*",
        "ext-curl": "*",
        "ext-dom": "*",
        "ext-gd": "*",
        "ext-json": "*",
        "ext-mbstring": "*",
        "ext-openssl": "*",
        "ext-pgsql": "*",
        "ext-tokenizer": "*",
        "ext-xml": "*",
        "ext-zip": "*",
        "laravel/framework": "5.7.*"
    },
    "require-dev": {
        "barryvdh/laravel-ide-helper": "^2.5",
        "filp/whoops": "^2.3",
        "friendsofphp/php-cs-fixer": "^2.15",
        "fzaninotto/faker": "^1.8",
        "itsgoingd/clockwork": "^3.1",
        "laravel/dusk": "^5.1.1",
        "mockery/mockery": "^1.2",
        "nunomaduro/larastan": "^0.3.15",
        "phpunit/phpunit": "^7.5",
        "themsaid/laravel-mail-preview": "^2.0"
    },
    "autoload": {
        "classmap": [
            "database"
        ],
        "psr-4": {
            "App\\": "app/"
        },
        "files": [
            "app/helpers.php"
        ]
    },
    "autoload-dev": {
        "psr-4": {
            "Tests\\": "tests/"
        }
    },
    "scripts": {
        "post-root-package-install": [
            "php -r \"copy('.env.example', '.env');\""
        ],
        "post-create-project-cmd": [
            "php artisan key:generate"
        ],
        "post-install-cmd": [
            "Illuminate\\Foundation\\ComposerScripts::postInstall"
        ],
        "post-update-cmd": [
            "Illuminate\\Foundation\\ComposerScripts::postUpdate"
        ],
        "post-autoload-dump": [
            "Illuminate\\Foundation\\ComposerScripts::postAutoloadDump",
            "@php artisan package:discover"
        ]
    },
    "config": {
        "preferred-install": "dist",
        "sort-packages": true
    }
}

Output of composer diagnose:

Checking composer.json: OK
Checking platform settings: OK
Checking git settings: OK
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com oauth access: OK
Checking disk free space: OK
Checking pubkeys: 
Tags Public Key Fingerprint: 57815BA2 7E54DC31 7ECC7CC5 573090D0  87719BA6 8F3BB723 4E5D42D0 84A14642
Dev Public Key Fingerprint: 4AC45767 E5EC2265 2F0C1167 CBBB8A2B  0C708369 153E328C AD90147D AFE50952
OK
Checking composer version: OK
Composer version: 1.9.0
PHP version: 7.2.19
PHP binary path: /usr/bin/php7.2

When I run this command: composer update all dependencies updated successfully and composer.lock file contains changes:

image

image

And I expected this to happen:
name\email\role keys order not changed; emoji not replaced with question mark.

@alcohol alcohol added the Support label Aug 8, 2019
@Seldaek

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

The author lines changing are an expected change since packagist.org had a change a few weeks ago and now stores the authors as they are defined in composer.json instead of normalizing them as it did before (which caused other issues). This should not change again after the first update changing them.

The emoji thing I am not sure what's up that seems wrong, I need to try it out.

@stloyd

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2019

IIRC there is issue within Docker on OSx that can lead to such replacement of emoji's in composer.lock file (can't remember what exactly failed, but it was something related to synchronization between host & docker containers).

AFAIK there is one workaround for this if this happens... hard one, but working (tested it as I also occurred this issue). You need to remove all vendor/ (i.e. via rm -rf vendor/*) folder on Docker container, reinstall dependencies & then update of composer.lock file should work.

ps. if you're not on osx... then dunno what the issue can be :)

@Seldaek

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

Indeed can not reproduce the emoji issue here, so I am going to assume it's an environment problem and not a Composer problem..

@Seldaek Seldaek closed this Aug 13, 2019
@UksusoFF

This comment has been minimized.

Copy link
Author

commented Aug 14, 2019

With emails normalize it's maybe packagist issue. Okay.
One times update not so mater.

But have trouble each time when composer updated on various machines. One machine use authors keys same as they described in vendor's composer.json but other always sort it as name/role/email/homepage.
Same with emoji: one machine make .lock with emoji from vendor's composer.json but other always replace with question mark.

It's really looks like environment trouble but if anyone have any guess where it's can be changed - please post to this issue.

@stloyd Yep. I run composer on Ubuntu 18.04 inside VirtualBox under MacOS. If you recall any details - please post to this issue.
I already do rm -rf vendor/* many times - nothing changed.

@Seldaek

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

Have you tried to rm -rf vendor/* on all machines involved, and then run composer install on them? Then they should get the data from lock file in vendor dir and should stop this ping-pong behavior..

@UksusoFF

This comment has been minimized.

Copy link
Author

commented Aug 14, 2019

@Seldaek yes, on all removed too.

@greysteil

This comment has been minimized.

Copy link
Contributor

commented Aug 26, 2019

We're seeing the emoji removal problem with Dependabot (fresh linux boxes without a vendor directory), and I get it when running on MacOS, too, using both Composer 1.9.0 and Composer 1.8.3 (clearing cache inbetween).

@greysteil

This comment has been minimized.

Copy link
Contributor

commented Aug 26, 2019

For a reproduction case, a Dependabot user put together https://github.com/b1rdex/dependabot-composer-bug - running composer update beberlei/assert --no-autoloader --no-scripts will generate a lockfile with the emoji issue. (It can probably be slimmed down a lot...)

@b1rdex

This comment has been minimized.

Copy link

commented Aug 27, 2019

But have trouble each time when composer updated on various machines. One machine use authors keys same as they described in vendor's composer.json but other always sort it as name/role/email/homepage.

Same for me. If it's packagist.org related, then why it's not consistent between different machines/envs?

@Seldaek could you please provide some debug instructions to dig this issue? As @greysteil mentioned above, I have a constantly reproducible example of the issue with reordering and emojis on the dependabot docker environment.

@Seldaek

This comment has been minimized.

Copy link
Member

commented Aug 29, 2019

All I see when trying to reproduce the case from @greysteil is the author email/role get flipped, which I fixed now in composer/packagist@1e96a08 by making this deterministic. It'll take a while to resolve itself across the board though as packages need to update.

I still don't see a problem with emojis.

@greysteil

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2019

You're not seeing this line change?

@Seldaek

This comment has been minimized.

Copy link
Member

commented Aug 29, 2019

Nope I don't. It's 🔍 before and after the update.

@greysteil

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2019

Drat - in that case I have nothing to add on reproduction steps. 😢

Thanks for looking into it.

@b1rdex

This comment has been minimized.

Copy link

commented Aug 30, 2019

Ok, guys, it looks like I've managed to reproduce the issue. Dependabot Dockerfile has LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8" instructions and locale-gen en_US.UTF-8. AFAIR, MacOS has locales set to UTF-8 too. And it seems to be the cause of the issue. So I set these env variables and generated locales on docker, removed vendor/ folder and emojis got corrupted after composer update.

I added Dockerfile and instructions to the repo with the bug: https://github.com/b1rdex/dependabot-composer-bug/blob/master/README.md

🔍 in nette/finder changes after composer update -v beberlei/assert --no-autoloader --no-scripts --ignore-platform-reqs ran in docker env with UTF-8 locales.

@Seldaek, please give it a try.

@greysteil is there a real need for LC_ALL="en_US.UTF-8" LANG="en_US.UTF-8" instructions and locale-gen en_US.UTF-8 in dependabot? Could we drop them?

@Seldaek

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

Using UTF-8 sounds like a good thing here tbh, and I have LANG=en_US.UTF-8 as well on my local env. I am not going to dig around docker trying to figure out what is going on here though, sorry.. If json_encode + file_put_contents breaks on your env I don't know what is off but that's about all we do there so I don't see how it would break.

@morozov

This comment has been minimized.

Copy link

commented Oct 7, 2019

FWIW, in my case, removing the vendor directory did solve the problem, so it looks like the characters don't get corrupted during the update but instead are already stored corrupted under vendor/ and are taken from there (unfortunately, I don't have a backup to confirm that).

Before I removed the vendor directory, I also tried running the update using the official docker image and the issue did reproduce as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.