lock file prevents install with --version of plugins that are needed by others

[jsvd]

In a release, the .lock file will contain the reference to a plugin's locked version even after being uninstalled IF other plugins depend on it. Example:

/tmp % tar -zxf logstash-2.0.0.dev.tar.gz
/tmp % cd logstash-2.0.0.dev
/tmp/logstash-2.0.0.dev % grep "logstash-codec-json .*0" Gemfile.jruby-1.9.lock
    logstash-codec-json (0.1.7)
/tmp/logstash-2.0.0.dev % bin/plugin uninstall logstash-codec-json
Uninstalling logstash-codec-json
/private/tmp/logstash-2.0.0.dev/vendor/local_gems/fccdd8d6/logstash-core-2.0.0.dev-java/lib/logstash/version.rb:3 warning: already initialized constant LOGSTASH_VERSION
/tmp/logstash-2.0.0.dev % grep "logstash-codec-json .*0" Gemfile.jruby-1.9.lock
    logstash-codec-json (0.1.7)
/tmp/logstash-2.0.0.dev % bin/plugin install --version="0.1.6" logstash-codec-json
Validating logstash-codec-json-0.1.6
Installing logstash-codec-json
Plugin not found, aborting
ERROR: Installation Aborted, message: You have requested:
  logstash-codec-json = 0.1.6

The bundle currently has logstash-codec-json locked at 0.1.7.
Try running `bundle update logstash-codec-json`
/tmp/logstash-2.0.0.dev %

[jsvd]

Also a simple uninstall of a plugin that others depend on "fails":

% bin/plugin list tcp
logstash-input-tcp
logstash-output-tcp
% bin/plugin uninstall logstash-input-tcp
Uninstalling logstash-input-tcp
% bin/plugin list tcp
logstash-input-tcp
logstash-output-tcp
% bin/plugin uninstall logstash-input-tcp
ERROR: Uninstall Aborted, message: This plugin has not been previously installed, aborting

[jsvd]

Also a simple uninstall of a plugin that others depend on "fails":

% bin/plugin list tcp
logstash-input-tcp
logstash-output-tcp
% bin/plugin uninstall logstash-input-tcp
Uninstalling logstash-input-tcp
% bin/plugin list tcp
logstash-input-tcp
logstash-output-tcp
% bin/plugin uninstall logstash-input-tcp
ERROR: Uninstall Aborted, message: This plugin has not been previously installed, aborting

[purbon]

I confirm this issue.

[purbon]

I confirm this issue.

[jsvd]

The uninstalled plugin remains in vendor/bundle:

 % find . -name *logstash-input-tcp* 
./vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5
./vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/logstash-input-tcp.gemspec
./vendor/bundle/jruby/1.9/specifications/logstash-input-tcp-0.1.5.gemspec

And while it's removed from Gemfile, it's still present in the .lock:

% grep "input-tcp" Gemfile*
Gemfile.jruby-1.9.lock:      logstash-input-tcp
Gemfile.jruby-1.9.lock:    logstash-input-tcp (0.1.5)

[jsvd]

The uninstalled plugin remains in vendor/bundle:

 % find . -name *logstash-input-tcp* 
./vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5
./vendor/bundle/jruby/1.9/gems/logstash-input-tcp-0.1.5/logstash-input-tcp.gemspec
./vendor/bundle/jruby/1.9/specifications/logstash-input-tcp-0.1.5.gemspec

And while it's removed from Gemfile, it's still present in the .lock:

% grep "input-tcp" Gemfile*
Gemfile.jruby-1.9.lock:      logstash-input-tcp
Gemfile.jruby-1.9.lock:    logstash-input-tcp (0.1.5)

[colinsurprenant]

is that a regression from rc3

[colinsurprenant]

is that a regression from rc3

[jsvd]

This is present in rc2 and rc3 as well, haven't tested rc1..

[jsvd]

This is present in rc2 and rc3 as well, haven't tested rc1..

[colinsurprenant]

ok, so I am "relieved" that this is not a #3111 regression and we should decided if we make that a rc4 blocker or not.

[colinsurprenant]

ok, so I am "relieved" that this is not a #3111 regression and we should decided if we make that a rc4 blocker or not.

[colinsurprenant]

Ok, here's the thing: it is totally normal that a plugin, or gem, that is a dependency of another plugin/gem is not uninstalled we asked to do so. To be able to remove it you'll have to remove the plugin/gem that depends on the one you are trying to remove.

I would even add: thanks to bundler for doing that, it avoid potential problem where if you uninstall a dependency, the plugin/gem that depend on it will obviously fail. If you were able to uninstall logstash-input-tcp using logstash-input-syslog would crash.

So, I am not sure how to deal with this, but for me it's more a UI problem. could we check that and report a warning to the user saying: sorry, impossible to uninstall ?

[colinsurprenant]

Ok, here's the thing: it is totally normal that a plugin, or gem, that is a dependency of another plugin/gem is not uninstalled we asked to do so. To be able to remove it you'll have to remove the plugin/gem that depends on the one you are trying to remove.

I would even add: thanks to bundler for doing that, it avoid potential problem where if you uninstall a dependency, the plugin/gem that depend on it will obviously fail. If you were able to uninstall logstash-input-tcp using logstash-input-syslog would crash.

So, I am not sure how to deal with this, but for me it's more a UI problem. could we check that and report a warning to the user saying: sorry, impossible to uninstall ?

[ph]

Keep in mind some plugins depends on other plugins only to run their tests but this dependencies is specified as a runtime dependency and not a development dependency in their gemspec. There is an open issue to clear them in #2078.

[ph]

Keep in mind some plugins depends on other plugins only to run their tests but this dependencies is specified as a runtime dependency and not a development dependency in their gemspec. There is an open issue to clear them in #2078.

[suyograo]

How about adding a --force option to remove all associated dependencies and plugins by walking the dependency tree? Not sure if this dependency graph/API is available, but just throwing it out there.

[suyograo]

How about adding a --force option to remove all associated dependencies and plugins by walking the dependency tree? Not sure if this dependency graph/API is available, but just throwing it out there.

[jsvd]

I agree that uninstall should not work if other gems depend on it. "can't uninstall this due to dependencies" seems to be the quickest and still sane fix to this situation, as @colinsurprenant suggested.
The current behaviour just isn't right:

% bin/plugin uninstall logstash-input-tcp
Uninstalling logstash-input-tcp
% bin/logstash -e 'input { tcp { port => 3333} } filter { split {} mutate { remove_field => [ "message" ] } } output { stdout { codec => rubydebug } }'
Logstash startup completed
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed
% bin/plugin uninstall logstash-input-tcp
ERROR: Uninstall Aborted, message: This plugin has not been previously installed, aborting
% bin/logstash -e 'input { tcp { port => 3333} } filter { split {} mutate { remove_field => [ "message" ] } } output { stdout { codec => rubydebug } }'
Logstash startup completed
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed

[jsvd]

I agree that uninstall should not work if other gems depend on it. "can't uninstall this due to dependencies" seems to be the quickest and still sane fix to this situation, as @colinsurprenant suggested.
The current behaviour just isn't right:

% bin/plugin uninstall logstash-input-tcp
Uninstalling logstash-input-tcp
% bin/logstash -e 'input { tcp { port => 3333} } filter { split {} mutate { remove_field => [ "message" ] } } output { stdout { codec => rubydebug } }'
Logstash startup completed
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed
% bin/plugin uninstall logstash-input-tcp
ERROR: Uninstall Aborted, message: This plugin has not been previously installed, aborting
% bin/logstash -e 'input { tcp { port => 3333} } filter { split {} mutate { remove_field => [ "message" ] } } output { stdout { codec => rubydebug } }'
Logstash startup completed
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed

[purbon]

After talking to @jsvd I removed my comment as It was pointless.

[purbon]

After talking to @jsvd I removed my comment as It was pointless.

[ph]

Maybe give the user a nice warning message like this?

(taken from the gem)

You have requested to uninstall the gem:
    arr-pm-0.0.9

fpm-1.3.3 depends on arr-pm (~> 0.0.9)
If you remove this gem, these dependencies will not be met.
Continue with Uninstall? [yN]

[ph]

Maybe give the user a nice warning message like this?

(taken from the gem)

You have requested to uninstall the gem:
    arr-pm-0.0.9

fpm-1.3.3 depends on arr-pm (~> 0.0.9)
If you remove this gem, these dependencies will not be met.
Continue with Uninstall? [yN]

[purbon]

+1 on nice warnings 👍 feedback is always nice.

[purbon]

+1 on nice warnings 👍 feedback is always nice.

[jordansissel]

Some users on IRC are reporting this as well

[jordansissel]

Some users on IRC are reporting this as well

[cdenneen]

understood this is outstanding bug but what is a work around for now?
1.5.4 shipped with mutate locked to 1.0.1 which has a bug in it.
Need to install 1.0.2 but lockfile isn't allowing.

[cdenneen]

understood this is outstanding bug but what is a work around for now?
1.5.4 shipped with mutate locked to 1.0.1 which has a bug in it.
Need to install 1.0.2 but lockfile isn't allowing.

[purbon]

@cdenneen if you take a look at the .lock file, the only plugin that has mutate as dependency is the multiline filter, a good workaround to fix this for now would be to uninstall the multiline, then update the mutate filter and so on in case you like to have multiline back. hope this helps.

That multiline has as a runtime dependency mutate, makes no sense. Will fix this now.

[purbon]

@cdenneen if you take a look at the .lock file, the only plugin that has mutate as dependency is the multiline filter, a good workaround to fix this for now would be to uninstall the multiline, then update the mutate filter and so on in case you like to have multiline back. hope this helps.

That multiline has as a runtime dependency mutate, makes no sense. Will fix this now.

[cdenneen]

Wouldn't the lock file dependency be better using ~> 1.0.1 syntax so all 1.0.x updates are permitted like 1.0.2?

[cdenneen]

Wouldn't the lock file dependency be better using ~> 1.0.1 syntax so all 1.0.x updates are permitted like 1.0.2?

[purbon]

I amend my previous messages, this is still an issue.

[purbon]

I amend my previous messages, this is still an issue.

[purbon]

A feedback loop has been proposed in #4288, this does not fix the issue per se, but provides feedback to users about the dependency tree involved when trying to uninstall a plugin.

[purbon]

A feedback loop has been proposed in #4288, this does not fix the issue per se, but provides feedback to users about the dependency tree involved when trying to uninstall a plugin.