New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spec-mount rest-backend-config does not work any more #1633

Closed
BernhardDenner opened this Issue Oct 21, 2017 · 9 comments

Comments

Projects
None yet
2 participants
@BernhardDenner
Contributor

BernhardDenner commented Oct 21, 2017

Describe what you wanted to do

prepare the rest-backend configuration with kdb mount-rest-backend-config, which involves

kdb mount rest-backend-spec.ini spec/sw/elektra/restbackend/#0/ ni
kdb spec-mount /sw/elektra/restbackend/#0/ ni

This leads to

Got mountpoint from spec/sw/elektra/restbackend/#0/current with 33 nodes, configfile:  and mountpoint: /sw/elektra/restbackend/#0/current/backend/api/description/html
The command kdb spec-mount terminated unsuccessfully with the info:
The path you entered is invalid.
Try to add another path instead.

Filenames are typically (depending on
resolver) not allowed to contain '..'.

For more information see:
 kdb info <your resolver>
Please report the issue at https://issues.libelektra.org/

Describe what you expected

rest-backend-config.ini mounted according to defined specification.

Describe what actually happened

The command kdb spec-mount /sw/elektra/restbackend/#0/ ni gives the above error message.

Steps to reproduce:

# in case you do not have rest-backend compiled
cp <libelektra-src>/src/tools/rest-backend/config-specifiction.ini.in /usr/local/share/elektra/specification/rest-backend-config.ini
# manually substitute cmake vars
vim /usr/local/share/elektra/specification/rest-backend-config.ini
kdb mount rest-backend-config.ini spec/sw/elektra/restbackend/#0 ni
kdb spec-mount /sw/elektra/restbackend/#0

The elektra version GIT sha1: e5624dd (master from end of May) works as expected.

System Information

  • Elektra Version: master
  • verified on Debian 9 and Ubuntu 16.04

@BernhardDenner BernhardDenner added the bug label Oct 21, 2017

@BernhardDenner

This comment has been minimized.

Show comment
Hide comment
@BernhardDenner

BernhardDenner Oct 21, 2017

Contributor

Some debugging I did:

The comand kdb ls spec/sw/elektra/restbackend gives

with the current master build (spec-mount not working)

/sw/elektra/restbackend/#0/current/backend/api/description/html
...
spec/sw/elektra/restbackend/#0/current
...

while the old build (from mentioned GIT hash - spec-mount working)

spec/sw/elektra/restbackend/#0/current
...

So, now the kdb ls contains cascading keys too.
I've done a short verification if this triggers the bug with the help of the Ruby bindings

require 'kdbtools'

db = Kdb.open
ks = Kdb::KeySet.new
db.get ks, 'spec/sw/elektra/restbackend/#0'
ks.pretty_print
spr = Kdbtools::SpecReader.new
spr.read_specification ks
m = spr.get_backends
b = m.first[1]
b.nodes             # => 33
b.get_config_file   # => ''
b.get_mountpoint    # => /sw/elektra/restbackend/#0/current/backend/api/description/html

# now create a KeySet without cascading keys
ks_spec = Kdb::KeySet.new(ks.find_all {|x| x.name =~ /^spec/})
spr = Kdbtools::SpecReader.new
spr.read_specification ks_spec
m = spr.get_backends
b = m.first[1]
# those are the values reported by a working kdb spec-mount -v ...
b.nodes             # => 45
b.get_config_file   # => 'rest-backend.ini'
b.get_mountpoint    # => /sw/elektra/restbackend/#0/current
Contributor

BernhardDenner commented Oct 21, 2017

Some debugging I did:

The comand kdb ls spec/sw/elektra/restbackend gives

with the current master build (spec-mount not working)

/sw/elektra/restbackend/#0/current/backend/api/description/html
...
spec/sw/elektra/restbackend/#0/current
...

while the old build (from mentioned GIT hash - spec-mount working)

spec/sw/elektra/restbackend/#0/current
...

So, now the kdb ls contains cascading keys too.
I've done a short verification if this triggers the bug with the help of the Ruby bindings

require 'kdbtools'

db = Kdb.open
ks = Kdb::KeySet.new
db.get ks, 'spec/sw/elektra/restbackend/#0'
ks.pretty_print
spr = Kdbtools::SpecReader.new
spr.read_specification ks
m = spr.get_backends
b = m.first[1]
b.nodes             # => 33
b.get_config_file   # => ''
b.get_mountpoint    # => /sw/elektra/restbackend/#0/current/backend/api/description/html

# now create a KeySet without cascading keys
ks_spec = Kdb::KeySet.new(ks.find_all {|x| x.name =~ /^spec/})
spr = Kdbtools::SpecReader.new
spr.read_specification ks_spec
m = spr.get_backends
b = m.first[1]
# those are the values reported by a working kdb spec-mount -v ...
b.nodes             # => 45
b.get_config_file   # => 'rest-backend.ini'
b.get_mountpoint    # => /sw/elektra/restbackend/#0/current
@markus2330

This comment has been minimized.

Show comment
Hide comment
@markus2330

markus2330 Oct 21, 2017

Contributor

Hi, I see several problems in your commands, the third problem is the most likely the actual cause:

  1. You use config-specification.ini from the src dir, but you should use the one from the build dir. You might have wrongly substituted values. Correct copy command for me is: cp ~build/src/tools/rest-backend/config-specification.ini /usr/local/share/elektra/specification/rest-backend-config.ini
  2. The path you entered is non-default: It is /usr/local/share/elektra/specification/ (without s at the end, see above).
  3. The specification file needs to be mounted with the ni plugin or ini plugin with meta option. Correct mount command would be: kdb mount rest-backend-config.ini spec/sw/elektra/restbackend/#0 ni

With these three changes kdb spec-mount /sw/elektra/restbackend/#0 worked without errors for me. If this does not solve the issue for you, please post the content of config-specification.in and the output of kdb export spec/sw/elektra/restbackend/#0 ni.

Please keep the issue open in any case: I agree that the error message is not helpful. It should be something like: metadata "mountpoint" not found or even give tips that the INI needs to be mounted with the "meta" option.

Contributor

markus2330 commented Oct 21, 2017

Hi, I see several problems in your commands, the third problem is the most likely the actual cause:

  1. You use config-specification.ini from the src dir, but you should use the one from the build dir. You might have wrongly substituted values. Correct copy command for me is: cp ~build/src/tools/rest-backend/config-specification.ini /usr/local/share/elektra/specification/rest-backend-config.ini
  2. The path you entered is non-default: It is /usr/local/share/elektra/specification/ (without s at the end, see above).
  3. The specification file needs to be mounted with the ni plugin or ini plugin with meta option. Correct mount command would be: kdb mount rest-backend-config.ini spec/sw/elektra/restbackend/#0 ni

With these three changes kdb spec-mount /sw/elektra/restbackend/#0 worked without errors for me. If this does not solve the issue for you, please post the content of config-specification.in and the output of kdb export spec/sw/elektra/restbackend/#0 ni.

Please keep the issue open in any case: I agree that the error message is not helpful. It should be something like: metadata "mountpoint" not found or even give tips that the INI needs to be mounted with the "meta" option.

@BernhardDenner

This comment has been minimized.

Show comment
Hide comment
@BernhardDenner

BernhardDenner Oct 21, 2017

Contributor

Ok, for your three points, that are really errors in my bug report. I've cross-checked it now with my terminal history:

You use config-specification.ini from the src dir, but you should use the one from the build dir. You might have wrongly substituted values. Correct copy command for me is: cp ~build/src/tools/rest-backend/config-specification.ini /usr/local/share/elektra/specification/rest-backend-config.ini

The substitution was just mentioned to reproduce the problem without requiring to build the rest-backend (having to install its deps). On the REST-backend target system I've used the generated one.

The path you entered is non-default: Its /usr/local/share/elektra/specification/ (without s at the end, see above).

That was a mistake in my bug report, of course it is /usr/local/share/elektra/specification/

The specification file needs to be mounted with the ni plugin or ini plugin with meta option. Correct mount command would be: kdb mount rest-backend-config.ini spec/sw/elektra/restbackend/#0 ni

Same here: I've forgotten to add the ni in my bug report. I've done the mount of the spec file with the ni plugin.

I've updated my bug report accordingly.
The problem still exists for me. I've done all my tests on newly installed systems (within a container). So no existing elektra configuration/file/mounts existed before.

I've attached my results of kdb export spec/sw/elektra/restbackend/#0 ni
export_spec_restbackend_ni.ini.gz

The used config-specification.ini.in is exactly the one found in this repo config-specification.ini.in

On Ubuntu Elektra was built using the following command

export LANG=en_US.UTF-8 && \
	mkdir /root/build && cd /root/build && \
	cmake -DBINDINGS=ALL -DPLUGINS=ALL /root/libelektra && \
	make -j4 && \
	make run_all || true && \
	make install && \
	ldconfig

On the Debian 9 system, I've used the scripts/build-homepage script.

Contributor

BernhardDenner commented Oct 21, 2017

Ok, for your three points, that are really errors in my bug report. I've cross-checked it now with my terminal history:

You use config-specification.ini from the src dir, but you should use the one from the build dir. You might have wrongly substituted values. Correct copy command for me is: cp ~build/src/tools/rest-backend/config-specification.ini /usr/local/share/elektra/specification/rest-backend-config.ini

The substitution was just mentioned to reproduce the problem without requiring to build the rest-backend (having to install its deps). On the REST-backend target system I've used the generated one.

The path you entered is non-default: Its /usr/local/share/elektra/specification/ (without s at the end, see above).

That was a mistake in my bug report, of course it is /usr/local/share/elektra/specification/

The specification file needs to be mounted with the ni plugin or ini plugin with meta option. Correct mount command would be: kdb mount rest-backend-config.ini spec/sw/elektra/restbackend/#0 ni

Same here: I've forgotten to add the ni in my bug report. I've done the mount of the spec file with the ni plugin.

I've updated my bug report accordingly.
The problem still exists for me. I've done all my tests on newly installed systems (within a container). So no existing elektra configuration/file/mounts existed before.

I've attached my results of kdb export spec/sw/elektra/restbackend/#0 ni
export_spec_restbackend_ni.ini.gz

The used config-specification.ini.in is exactly the one found in this repo config-specification.ini.in

On Ubuntu Elektra was built using the following command

export LANG=en_US.UTF-8 && \
	mkdir /root/build && cd /root/build && \
	cmake -DBINDINGS=ALL -DPLUGINS=ALL /root/libelektra && \
	make -j4 && \
	make run_all || true && \
	make install && \
	ldconfig

On the Debian 9 system, I've used the scripts/build-homepage script.

@markus2330

This comment has been minimized.

Show comment
Hide comment
@markus2330

markus2330 Oct 21, 2017

Contributor

Okay, thank you, now its more clear. I'll need a Debian 9 to reproduce.

Can you bisect which commit introduces the issue? (Have you automated the creation of containers?)

As workaround you can simply not use kdb spec-mount, in the end kdb spec-mount also does nothing but a kdb mount (with the collected plugins as arguments). But it seems to be a serious bug, we should fix it for the release. Maybe something is even broken in the resolver? A good way forward would be to know which commit caused the problem.

I've done a short verification if this triggers the bug with the help of the Ruby bindings

And, did it?

Contributor

markus2330 commented Oct 21, 2017

Okay, thank you, now its more clear. I'll need a Debian 9 to reproduce.

Can you bisect which commit introduces the issue? (Have you automated the creation of containers?)

As workaround you can simply not use kdb spec-mount, in the end kdb spec-mount also does nothing but a kdb mount (with the collected plugins as arguments). But it seems to be a serious bug, we should fix it for the release. Maybe something is even broken in the resolver? A good way forward would be to know which commit caused the problem.

I've done a short verification if this triggers the bug with the help of the Ruby bindings

And, did it?

@BernhardDenner

This comment has been minimized.

Show comment
Hide comment
@BernhardDenner

BernhardDenner Oct 21, 2017

Contributor

Can you bisect which commit introduces the issue? (Have you automated the creation of containers?)

I was already afraid you will ask for that ;)
Yes I can do that.

And, did it?

Yes, it did.

As far as I can tell: if SpecReader gets a keySet with the contained cascading keys it returns wrong values for the backend. Once the cascading keys are removed from the keySet SepcReader works as expected.

Contributor

BernhardDenner commented Oct 21, 2017

Can you bisect which commit introduces the issue? (Have you automated the creation of containers?)

I was already afraid you will ask for that ;)
Yes I can do that.

And, did it?

Yes, it did.

As far as I can tell: if SpecReader gets a keySet with the contained cascading keys it returns wrong values for the backend. Once the cascading keys are removed from the keySet SepcReader works as expected.

@markus2330

This comment has been minimized.

Show comment
Hide comment
@markus2330

markus2330 Oct 21, 2017

Contributor

I was already afraid you will ask for that ;)
Yes I can do that.

Thank you.

The problem might be related "spec: also create default keys" 670cd01 (or more precisely 427a40a). They look like cascading keys.

But from the error I was more thinking in the direction of 38e27eb (and other commits heavily changing the resolver).

Hopefully the commits help you to faster locate the problem.

What makes no sense for me is why it seems to be only reproducible for Debian 9. Did you globally mount spec (can you do kdb rm -r system/elektra/globalplugins/ or kdb global-mount and still reproduce)?

Contributor

markus2330 commented Oct 21, 2017

I was already afraid you will ask for that ;)
Yes I can do that.

Thank you.

The problem might be related "spec: also create default keys" 670cd01 (or more precisely 427a40a). They look like cascading keys.

But from the error I was more thinking in the direction of 38e27eb (and other commits heavily changing the resolver).

Hopefully the commits help you to faster locate the problem.

What makes no sense for me is why it seems to be only reproducible for Debian 9. Did you globally mount spec (can you do kdb rm -r system/elektra/globalplugins/ or kdb global-mount and still reproduce)?

@BernhardDenner

This comment has been minimized.

Show comment
Hide comment
@BernhardDenner

BernhardDenner Oct 21, 2017

Contributor

What makes no sense for me is why it seems to be only reproducible for Debian 9.

It's not a Debian9 only issue. I've reproduced this in Ubuntu 16.04 too.

Did you globally mount spec (can you do kdb rm -r system/elektra/globalplugins/ or kdb global-mount and still reproduce)?

I thought spec is mouted globally by default, isn't it?
system/elektra/globalplugins/ does not exits and kdb global-mount does not change any thing.
Doing a kdb global-mount spec results in

The command kdb global-mount terminated unsuccessfully with the info:
It is not allowed to insert the same plugin (spec#spec) again!
Try to add other plugins or other refnames (part after #) instead.
Please report the issue at https://issues.libelektra.org/
Contributor

BernhardDenner commented Oct 21, 2017

What makes no sense for me is why it seems to be only reproducible for Debian 9.

It's not a Debian9 only issue. I've reproduced this in Ubuntu 16.04 too.

Did you globally mount spec (can you do kdb rm -r system/elektra/globalplugins/ or kdb global-mount and still reproduce)?

I thought spec is mouted globally by default, isn't it?
system/elektra/globalplugins/ does not exits and kdb global-mount does not change any thing.
Doing a kdb global-mount spec results in

The command kdb global-mount terminated unsuccessfully with the info:
It is not allowed to insert the same plugin (spec#spec) again!
Try to add other plugins or other refnames (part after #) instead.
Please report the issue at https://issues.libelektra.org/
@markus2330

This comment has been minimized.

Show comment
Hide comment
@markus2330

markus2330 Oct 21, 2017

Contributor

It's not a Debian9 only issue. I've reproduced this in Ubuntu 16.04 too.

Yes, I only meant the hypothesis that the error could stem from the Debian versions: 8 seems to work and 9 fails. But maybe the Debian version has nothing to do with the issue and there is something else different on my system (which makes it work). It seems to not depend on the content of the KDB, I just retried after kdb stash.

I thought spec is mou[n]ted globally by default, isn't it?

Yes.

It would be great if you can incorporate your script (for the bisect) to the test cases afterwards (to avoid regressions).

Contributor

markus2330 commented Oct 21, 2017

It's not a Debian9 only issue. I've reproduced this in Ubuntu 16.04 too.

Yes, I only meant the hypothesis that the error could stem from the Debian versions: 8 seems to work and 9 fails. But maybe the Debian version has nothing to do with the issue and there is something else different on my system (which makes it work). It seems to not depend on the content of the KDB, I just retried after kdb stash.

I thought spec is mou[n]ted globally by default, isn't it?

Yes.

It would be great if you can incorporate your script (for the bisect) to the test cases afterwards (to avoid regressions).

@BernhardDenner

This comment has been minimized.

Show comment
Hide comment
@BernhardDenner

BernhardDenner Oct 21, 2017

Contributor

After lots of compiling, git bisect finally ended with the following result:

427a40a5e0ba710924d45eac995cfa89e6c682f2 is the first bad commit
commit 427a40a5e0ba710924d45eac995cfa89e6c682f2
Author: Thomas Waser <thomas.waser@libelektra.org>
Date:   Sun May 21 16:10:25 2017 +0200

    spec: also create default keys

:040000 040000 2c85c3820a58fd3905bb474863be562ff6306832 ba4a27d3c8886af12b301d9a23f82318dbfe1211 M	src
bisect run success

So it seems 427a40a introduced the issue, as you already guessed ;)

Contributor

BernhardDenner commented Oct 21, 2017

After lots of compiling, git bisect finally ended with the following result:

427a40a5e0ba710924d45eac995cfa89e6c682f2 is the first bad commit
commit 427a40a5e0ba710924d45eac995cfa89e6c682f2
Author: Thomas Waser <thomas.waser@libelektra.org>
Date:   Sun May 21 16:10:25 2017 +0200

    spec: also create default keys

:040000 040000 2c85c3820a58fd3905bb474863be562ff6306832 ba4a27d3c8886af12b301d9a23f82318dbfe1211 M	src
bisect run success

So it seems 427a40a introduced the issue, as you already guessed ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment