Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

composer update --no-dev loads dependencies from require-dev key #3742

Closed
internalsystemerror opened this issue Feb 13, 2015 · 29 comments

Comments

@internalsystemerror
Copy link

@internalsystemerror internalsystemerror commented Feb 13, 2015

From the output of composer help update...
--no-dev Disables installation of require-dev packages.

I'm using the following:

    "require": {
        "php": ">=5.4",
        "videlalvaro/php-amqplib": "2.*",
        "ebaye/ruc-config": "1.*",
        "ebaye/ruc-doctrine": "~1.1",
        "ebaye/ruc-translate": "1.*",
        "ebaye/ruc-logger": "1.*",
        "ebaye/ruc-transporter": "1.*",
        "ebaye/manhattan": "1.*"
    },
    "require-dev": {
        "ebaye/ruc-config": "dev-trunk",
        "ebaye/ruc-doctrine": "dev-trunk",
        "ebaye/ruc-translate": "dev-trunk",
        "ebaye/ruc-logger": "dev-trunk",
        "ebaye/ruc-transporter": "dev-trunk",
        "ebaye/manhattan": "dev-trunk"
    },

I remove the require-dev key, and do composer update... Then I re-add the require-dev key and run composer update --no-dev and I get the following output:

[dev@localhost ruc]$ composer update -o --no-dev
Loading composer repositories with package information
Updating dependencies
  - Updating ebaye/ruc-config (1.0.1 => dev-trunk)
    Checking out /trunk/@54

  - Updating ebaye/ruc-doctrine (1.1.1 => dev-trunk)
    Checking out /trunk/@63

  - Updating ebaye/ruc-translate (1.0.0 => dev-trunk)
    Checking out /trunk/@48

  - Updating ebaye/ruc-logger (1.0.4 => dev-trunk)
    Checking out /trunk/@262

  - Updating ebaye/manhattan (1.0.1 => dev-trunk)
    Checking out /trunk/@203

  - Updating ebaye/ruc-transporter (1.1.3 => dev-trunk)
    Checking out /trunk/@169

Writing lock file
Generating optimized autoload files

Am I doing something wrong? I'm using the latest version of composer at the time of writing.

EDIT: It may or may not be relevant that we're using SVN.

@internalsystemerror internalsystemerror changed the title composer update --no-dev loads installs from require-dev key composer update --no-dev loads depencies from require-dev key Feb 13, 2015
@internalsystemerror internalsystemerror changed the title composer update --no-dev loads depencies from require-dev key composer update --no-dev loads dependencies from require-dev key Feb 13, 2015
@sbuzonas

This comment has been minimized.

Copy link
Contributor

@sbuzonas sbuzonas commented Feb 15, 2015

That's interesting. I've never seen dev requirements replacing normal requirements before. That can only work if both constraints can be satisfied. I'm seeing similar behavior on git as well, but what I am noticing though is that the commit hash that is checked out is equivalent to the tagged version it would checkout if I did not have a development reference in my require-dev. Is the /trunk/@169 reference equivalent to /tags/1.1.3?

I think it might be happening somewhere in this method and ends up adding a second constraint to your package. I'm not too familiar with that portion of the installer or even what those forced updates are doing. But I can't get it to install something that does not satisfy my constraints in the normal requirements. The solver throws an exception if it can't satisfy both.

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 15, 2015

Ahhh I see what you're saying, that the revision is the same regardless as to whether its from the tag or the branch? I'll have to check this Monday as I don't have access outside the office. I am surprised that it is considering the require-dev key at all with a --no-dev flag. If the expected result is the same, however, then I can live with it. Thanks for the response, I'll post again Monday.

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 16, 2015

Ok, so unfortunately that's not what's happening... Upon checking the lock file, the only difference I see when running with and without the --no-dev flag is:

composer update
"packages-dev": []
composer update --no-dev
"packages-dev": null

The revisions etc are all the latest revision in dev-trunk. If I run --prefer-stable, then this has the desired outcome for now. I'm unable to replicate this situation using my own testing on git, I'll try to come up with the appropriate replication steps for others interested, but my suspicion is that this is Subversion related.

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 16, 2015

Unable to reproduce as those are private repositories I guess.

Can you try the following perhaps:

rm -rf vendor/ composer.lock
composer install --no-dev
composer update --no-dev

Does that also give you dev-trunk then?
Maybe related also, what is your minimum-stability set to in your root composer.json?

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 16, 2015

As we're committing the lock file, composer install --no-dev will read from whatever we commit. The idea is that during development we can run composer update, and then just before release, composer update --no-dev, and commit the lock file.

Running composer install --no-dev makes no difference with the following output:

[dev@localhost ruc]$ composer install --no-dev
Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files
[dev@localhost ruc]$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Nothing to install or update
Generating autoload files
[dev@localhost ruc]$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update
Generating autoload files
[dev@localhost ruc]$ composer update --no-dev
Loading composer repositories with package information
Updating dependencies
Nothing to install or update
Writing lock file
Generating autoload files

Yes these are private repositories using a satis packaging server. There is no minimum stability set currently. I've tried various forms of version constraint in the require-dev key, dev-trunk, 1.*@dev etc etc... all with the same results... --no-dev is not ignoring the require-dev key...

Snippet from my composer.json:

    "repositories": [
        {
            "packagist": false
        },
        {
            "type": "composer",
            "url": "http://packages.example.org" (edited for security)
        }
    ],
    "require": {
        "php": ">=5.4",
        "videlalvaro/php-amqplib": "2.*",
        "ebaye/ruc-config": "1.*",
        "ebaye/ruc-doctrine": "~1.1",
        "ebaye/ruc-translate": "1.*",
        "ebaye/ruc-logger": "1.*",
        "ebaye/ruc-transporter": "1.*",
        "ebaye/manhattan": "1.*"
    },
    "require-dev": {
        "ebaye/ruc-config": "1.*@dev",
        "ebaye/ruc-doctrine": "1.*@dev",
        "ebaye/ruc-translate": "1.*@dev",
        "ebaye/ruc-logger": "1.*@dev",
        "ebaye/ruc-transporter": "1.*@dev",
        "ebaye/manhattan": "1.*@dev"
    },

Each of the packages has their own composer.json and hosted from the same SVN server. Interestingly, I ran the following test to see the outcome...

In require-dev I added "phpunit/phpunit": "4.*", and added phpunit to our satis.json... The I get the following output:

[dev@localhost ruc]$ composer update -o
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Installing symfony/yaml (v2.6.4)
    Downloading: 100%         

  - Installing sebastian/version (1.0.4)
    Downloading: 100%         

  - Installing sebastian/global-state (1.0.0)
    Downloading: 100%         

  - Installing sebastian/recursion-context (1.0.0)
    Downloading: 100%         

  - Installing sebastian/exporter (1.2.0)
    Downloading: 100%         

  - Installing sebastian/environment (1.2.1)
    Downloading: 100%         

  - Installing sebastian/diff (1.2.0)
    Downloading: 100%         

  - Installing sebastian/comparator (1.1.1)
    Downloading: 100%         

  - Installing phpunit/php-text-template (1.2.0)
    Downloading: 100%         

  - Installing doctrine/instantiator (1.0.4)
    Downloading: 100%         

  - Installing phpunit/phpunit-mock-objects (2.3.0)
    Downloading: 100%         

  - Installing phpunit/php-timer (1.0.5)
    Downloading: 100%         

  - Installing phpunit/php-file-iterator (1.3.4)
    Downloading: 100%         

  - Installing phpunit/php-token-stream (1.4.0)
    Downloading: 100%         

  - Installing phpunit/php-code-coverage (2.0.15)
    Downloading: 100%         

  - Installing phpdocumentor/reflection-docblock (2.0.4)
    Downloading: 100%         

  - Installing phpspec/prophecy (v1.3.1)
    Downloading: 100%         

  - Installing phpunit/phpunit (4.5.0)
    Downloading: 100%         

sebastian/global-state suggests installing ext-uopz (*)
phpdocumentor/reflection-docblock suggests installing dflydev/markdown (~1.0)
phpdocumentor/reflection-docblock suggests installing erusev/parsedown (~1.0)
phpunit/phpunit suggests installing phpunit/php-invoker (~1.1)
Writing lock file
Generating optimized autoload files
[dev@localhost ruc]$ composer update -o --no-dev
Loading composer repositories with package information
Updating dependencies
  - Removing phpunit/phpunit (4.5.0)
  - Removing phpspec/prophecy (v1.3.1)
  - Removing phpdocumentor/reflection-docblock (2.0.4)
  - Removing phpunit/php-code-coverage (2.0.15)
  - Removing phpunit/php-token-stream (1.4.0)
  - Removing phpunit/php-file-iterator (1.3.4)
  - Removing phpunit/php-timer (1.0.5)
  - Removing phpunit/phpunit-mock-objects (2.3.0)
  - Removing doctrine/instantiator (1.0.4)
  - Removing phpunit/php-text-template (1.2.0)
  - Removing sebastian/comparator (1.1.1)
  - Removing sebastian/diff (1.2.0)
  - Removing sebastian/environment (1.2.1)
  - Removing sebastian/exporter (1.2.0)
  - Removing sebastian/recursion-context (1.0.0)
  - Removing sebastian/global-state (1.0.0)
  - Removing sebastian/version (1.0.4)
  - Removing symfony/yaml (v2.6.4)
Writing lock file
Generating optimized autoload files
[dev@localhost ruc]$ 

TLDR;

The issue only occurs when the same package is named in both the require and require-dev keys.

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 16, 2015

I explicitly added the rm command for a reason. You clearly did not execute it. Can I again ask that you run the steps I provided (all of them) and let us know the result?

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 16, 2015

Also, I think you don't understand the concept of require-dev. These should be extra dependencies you rely on during development (e.g. phpunit/phpunit). They should not be the same dependencies as specified in require. That doesn't make any sense.

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 16, 2015

Yes I missed off the rm in my post as I edited the output to display the relevant package information without exposing anything that may be confidential. The commands were still run as you originally specified, along with many other combinations in order to try and diagnose this issue.

For our use case this makes complete sense... You can run composer install --no-dev in production, and composer update in development. The point is that we can be working on multiple packages, so in development we want to be using the latest from dev-trunk, whereas in production we only want the tagged releases.

As stated in my original post, I'm basing this off what composer itself tells me...
--no-dev Disables installation of require-dev packages.

Is it mentioned anywhere in the documentation that different branches of the same package cannot be in both require AND require-dev?

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 16, 2015

[dev@localhost ruc]$ composer install --no-dev
Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating autoload files

This clearly shows you did not run the rm command.

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 16, 2015

Actually that shows that I did not SHOW that I ran the rm command, as said, I've done this.

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 16, 2015

If you ran the rm command, then your .lock file would not exist and it would not be installing from lock. So please, are you still going to insist that you did run it?

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 16, 2015

Ok, as I stated, I HAVE run those commands, though that output doesnt show it. If you meant to say, can I request the direct output from those commands, then please find attached below... Please note that I have to remove some packages from displaying, but that these are not included in the require-dev.

[dev@localhost ruc]$ rm -rf vendor composer.lock
[dev@localhost ruc]$ composer install --no-dev
Loading composer repositories with package information
Installing dependencies
  - Installing ebaye/ruc-config (dev-trunk)
    Checking out /trunk/@55

  - Installing ebaye/ruc-doctrine (dev-trunk)
    Checking out /trunk/@64

  - Installing ebaye/ruc-translate (dev-trunk)
    Checking out /trunk/@50

  - Installing ebaye/ruc-logger (dev-trunk)
    Checking out /trunk/@264

  - Installing ebaye/manhattan (dev-trunk)
    Checking out /trunk/@209

  - Installing ebaye/ruc-transporter (dev-trunk)
    Checking out /trunk/@171

Writing lock file
Generating autoload files
[dev@localhost ruc]$ rm -rf vendor composer.lock
[dev@localhost ruc]$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev)
  - Installing ebaye/ruc-config (dev-trunk)
    Checking out /trunk/@55

  - Installing ebaye/ruc-doctrine (dev-trunk)
    Checking out /trunk/@64

  - Installing ebaye/ruc-translate (dev-trunk)
    Checking out /trunk/@50

  - Installing ebaye/ruc-logger (dev-trunk)
    Checking out /trunk/@264

  - Installing ebaye/manhattan (dev-trunk)
    Checking out /trunk/@209

  - Installing ebaye/ruc-transporter (dev-trunk)
    Checking out /trunk/@171

Writing lock file
Generating autoload files
[dev@localhost ruc]$ rm -rf vendor composer.lock
[dev@localhost ruc]$ composer update --no-dev
Loading composer repositories with package information
Updating dependencies
  - Installing ebaye/ruc-config (dev-trunk)
    Checking out /trunk/@55

  - Installing ebaye/ruc-doctrine (dev-trunk)
    Checking out /trunk/@64

  - Installing ebaye/ruc-translate (dev-trunk)
    Checking out /trunk/@50

  - Installing ebaye/ruc-logger (dev-trunk)
    Checking out /trunk/@264

  - Installing ebaye/manhattan (dev-trunk)
    Checking out /trunk/@209

  - Installing ebaye/ruc-transporter (dev-trunk)
    Checking out /trunk/@171

Writing lock file
Generating autoload files
[dev@localhost ruc]$ rm -rf vendor composer.lock
[dev@localhost ruc]$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Installing ebaye/ruc-config (dev-trunk)
    Checking out /trunk/@55

  - Installing ebaye/ruc-doctrine (dev-trunk)
    Checking out /trunk/@64

  - Installing ebaye/ruc-translate (dev-trunk)
    Checking out /trunk/@50

  - Installing ebaye/ruc-logger (dev-trunk)
    Checking out /trunk/@264

  - Installing ebaye/manhattan (dev-trunk)
    Checking out /trunk/@209

  - Installing ebaye/ruc-transporter (dev-trunk)
    Checking out /trunk/@171

Writing lock file
Generating autoload files

As you can see this is as above, with the exact same results. (revision numbers have changed as we've commited since I last ran these commands, however these revisions are still different from the tagged releases)

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 17, 2015

Not sure why it still says updating after you remove vendor/, it should fall back to installing. Though that could just be a matter of log output not properly checking conditions.

Puzzling issue. Hard to debug cause I cannot reproduce it.

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 17, 2015

The only thing I can think of is the following:

The moment you do an install or update with dev requires, it will merge require and require-dev. This means all your stable version constrains get overwritten with your dev constraints (because you use it in a way you are not supposed to use it). These are then installed and locked in your composer.lock file. Subsequent installs will use your lock file. Subsequent updates however, should not.. I think. No clue where it goes "wrong" though for you.

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 17, 2015

@alcohol Could you specify where it says in any of the composer documentation that require-dev shouldn't contain the same packages as require? You're asserting that we're "using it wrong", I'd like to know where the evidence for that is.

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 17, 2015

require-dev (root-only)#

Lists packages required for developing this package, or running tests, etc. The dev 
requirements of the root package are installed by default. Both install or update 
support the --no-dev option that prevents dev dependencies from being installed.

require-dev is merged with your require section. So if they contain the same packages, they would overrule those requirements and then what would be the whole point?

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 17, 2015

You're twisting the words of the documentation, I will simply refer to your own quote:

Both install or update support the --no-dev option that prevents dev dependencies from being installed.

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 17, 2015

I am not twisting anything. The merging is literally done in the code using array_merge(requires, dev-requires). I think you just don't understand the concept of managing dependencies. There's no point in continuing this discussion.

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 17, 2015

I'm asking based on the documentation, not the code. As a user of composer, not a developer, the documentation tells me that I can use the --no-dev key to disable installation of require-dev packages. Why is it even necessary to read the dev requirements at all when --no-dev has been specified?

I'm well aware of how to manage dependencies, as I'm also well aware that when you design software, the users will use it how they want. Telling them they're using it wrong, has never been a good approach. Altering the documentation to read more clearly to its purpose would be my suggestion. However, I fail to see a problem with allowing --no-dev to prevent development packages being loaded?

@sbuzonas

This comment has been minimized.

Copy link
Contributor

@sbuzonas sbuzonas commented Feb 17, 2015

It isn't explicitly documented, it's just something that isn't normally done. From an internal perspective as far as the implementation goes it's difficult to handle packages listed in both because the require-dev section is consulted to determine which packages are to be removed.

Standard practice in the composer community is that you list your runtime dependencies in require and you list your development/test dependencies in require-dev. The lock file is normally committed in the repo with the dev dependencies and on deployment it's simply an install --no-dev.

Your configuration isn't within the standard use case because your require-dev undoubtedly replaces your require section. Your scenario may done closer to the norm if you actually had a second composer.json with different requires. Such as a composer-dev.json and you can call COMPOSER=composer-dev.json composer update to get your development dependencies.

Regardless, I do believe there is a bug somewhere and it should at the very least satisfy both constraints.

@internalsystemerror

This comment has been minimized.

Copy link
Author

@internalsystemerror internalsystemerror commented Feb 18, 2015

Using a separate composer.json, may also suffice for our needs, however using --prefer-stable suits for now. Thankyou very much.

Getting people here introduced to composer was a start, I'm now trying to help them to be able to develop across multiple packages in way that suits their development approach. There may be better approaches to this, but for the purposes of this ticket, I'll keep it to the following...

What is the expected result of having the same package with different constraints in both require/require-dev? If a decision was made that this should be ignored, then maybe generate a warning when composer is run? Otherwise I'll start working on a PR if desired?

@alcohol

This comment has been minimized.

Copy link
Member

@alcohol alcohol commented Feb 18, 2015

There is no defined expected result I imagine as this described use-case deviates from the norm. Internally require and require-dev are array_merge()'d, with the latter overwriting the former. So technically, the expected result is that the version constraints of require-dev will overrule those of require.

The common use-case is that require-dev contains libraries only used during development and testing, such as codesniffer, phpunit, etc. For a production release you would then simply run

composer install --no-dev
git archive

for example (or something similar).

I still don't understand why a composer install --no-dev from scratch resulted in dev-trunk being installed in your case. The only time I had similar behaviour was when my latest stable tag pointed to the same commit as dev-master (using git though, not svn). But I think that was merely related to the way it displayed/determined the version, and got fixed in a patch a while ago (iirc).

@stof

This comment has been minimized.

Copy link
Contributor

@stof stof commented Feb 21, 2015

Having the same packages in both require and require-dev can create weird cases in Composer. If you validate your composer.json with composer validate, you will see that it complains about it.

@Seldaek

This comment has been minimized.

Copy link
Member

@Seldaek Seldaek commented Feb 27, 2016

Closing as it's an edge case using a combination of a few features we don't recommend using and then well it's been a year and nobody fixed it so unless we get an easy to reproduce case I doubt it'll ever get fixed.

@shiva17gupta

This comment has been minimized.

Copy link

@shiva17gupta shiva17gupta commented May 3, 2016

getting error

@Bilge

This comment has been minimized.

Copy link

@Bilge Bilge commented Jun 18, 2016

I still have this problem if require-dev contains extension dependencies.

@curry684

This comment has been minimized.

Copy link
Contributor

@curry684 curry684 commented Jun 19, 2016

Given that this issue is based on a 1.5 year old codebase and already closed - please open a new issue as it most likely is.

Also, please be more descriptive than "getting error" or the hardly more verbose "I have this problem when I have extension dependencies". If we have nothing to reproduce we have nothing to fix.

@curry684

This comment has been minimized.

Copy link
Contributor

@curry684 curry684 commented Jun 19, 2016

As confirmed in #5444 you are experiencing a different issue and this one can remain closed.

@roni-estein

This comment has been minimized.

Copy link

@roni-estein roni-estein commented Nov 8, 2019

It isn't explicitly documented, it's just something that isn't normally done. From an internal perspective as far as the implementation goes it's difficult to handle packages listed in both because the require-dev section is consulted to determine which packages are to be removed.

Standard practice in the composer community is that you list your runtime dependencies in require and you list your development/test dependencies in require-dev. The lock file is normally committed in the repo with the dev dependencies and on deployment it's simply an install --no-dev.

Your configuration isn't within the standard use case because your require-dev undoubtedly replaces your require section. Your scenario may done closer to the norm if you actually had a second composer.json with different requires. Such as a composer-dev.json and you can call COMPOSER=composer-dev.json composer update to get your development dependencies.

Regardless, I do believe there is a bug somewhere and it should at the very least satisfy both constraints.

Knowing you can have two composer.json files was very helpful. I'm just starting to separate out duplicated and local code into private packages on a dev machine for local (laravel) commands and testing. The --no-dev docs explanation is very confusing. Knowing you can simply have 2 composer files has been very helpful in solving this.

I would follow up with just adding a few paragraphs to the docs explaining the procedure and how it might be useful for example in a situation where you are developing packages alongside apps. Cheers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.