Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker 'network=host' doesn't start - too many positional options have been specified #4141

Open
CountZukula opened this issue Jan 24, 2019 · 8 comments
Assignees
Labels
Milestone

Comments

@CountZukula
Copy link

@CountZukula CountZukula commented Jan 24, 2019

This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.

  • I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): Docker version 3.0.1 / LATEST
Cluster size: Not relevant
OS (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu (Docker)

Hardware details (for performance issues) Delete if unneeded
Platform (physical/VM/cloud instance type/docker): docker, running on physical node managed on cluster
Hardware: sockets= cores=4 hyperthreading=no memory=16G
Disks: (SSD/HDD, count) SSD, 16GB

I am trying to run Scylla on multiple virtual nodes, on which I have administrator access. The nodes can ping each other and are available on the same network. Currently, I'm trying to get it up and running using Docker, as I have tried beforehand on my local machine (which seemed to be going fine).

Now, on one of the virtual nodes, I am executing the following (simplified command as this already triggers my problem):

sudo docker run --name scylla -it --network host scylladb/scylla

I'm getting the following error, resulting in Scylla not starting up (error: too many positional options have been specified on the command line):

running: (['/usr/lib/scylla/scylla_dev_mode_setup', '--developer-mode', '1'],)
running: (['/usr/lib/scylla/scylla_io_setup'],)
2019-01-24 13:31:09,884 CRIT Supervisor running as root (no user in config file)
2019-01-24 13:31:09,884 WARN Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing
2019-01-24 13:31:09,884 WARN Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing
2019-01-24 13:31:09,884 WARN Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing
2019-01-24 13:31:09,899 INFO RPC interface 'supervisor' initialized
2019-01-24 13:31:09,899 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2019-01-24 13:31:09,899 INFO supervisord started with pid 10
2019-01-24 13:31:10,901 INFO spawned: 'scylla-jmx' with pid 13
2019-01-24 13:31:10,904 INFO spawned: 'scylla' with pid 14
2019-01-24 13:31:10,906 INFO spawned: 'scylla-housekeeping' with pid 15
error: too many positional options have been specified on the command line

Try --help.
2019-01-24 13:31:11,018 INFO exited: scylla (exit status 2; not expected)
Connecting to http://localhost:10000
Starting the JMX server
JMX is enabled to receive remote connections on port: 7199
2019-01-24 13:31:12,369 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-24 13:31:12,371 INFO spawned: 'scylla' with pid 32
2019-01-24 13:31:12,371 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
error: too many positional options have been specified on the command line

Try --help.
2019-01-24 13:31:12,480 INFO exited: scylla (exit status 2; not expected)
2019-01-24 13:31:14,485 INFO spawned: 'scylla' with pid 34
error: too many positional options have been specified on the command line

Try --help.

When removing 'network host' from the docker command, however, Scylla starts as expected.

@CountZukula CountZukula changed the title Docker 'network=host' Docker 'network=host' doesn't start - too many positional options have been specified Jan 24, 2019
@slivne

This comment has been minimized.

Copy link
Contributor

@slivne slivne commented Jan 27, 2019

checked locally on my machine sudo docker run --name scylla -it --network host scylladb/scylla

works
latest is 3.0.1

maybe related to running on ubuntu - although its not clear how that can cause an issue.

@slivne

This comment has been minimized.

Copy link
Contributor

@slivne slivne commented Feb 7, 2019

#4203 would help to solve such issues

@tzemanovic

This comment has been minimized.

Copy link

@tzemanovic tzemanovic commented Feb 21, 2019

I'm hitting the same issue running Docker on mac

kbr- added a commit to kbr-/scylla that referenced this issue Jul 1, 2019
Fixes scylladb#4203 and scylladb#4141.
Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run.
The first parse is there to check if the "--version" flag is present --- in this case the version is printed
and the program exists.  However, most of the arguments were improperly treated as positional arguments
the first time they were parsed (e.g., "--network host" would treat "host" as a positional argument).
This commit fixes the issue and prints the parsed arguments.

Signed-off-by: Kamil Braun <kbraun@scylladb.com>
kbr- added a commit to kbr-/scylla that referenced this issue Jul 1, 2019
Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run.
The first parse is there to check if the "--version" flag is present --- in this case the version is printed
and the program exists.  The second parsing is correct; however, most of the arguments were improperly treated
as positional arguments during the first parsing (e.g., "--network host" would treat "host" as a positional argument).
This happened because the arguments weren't known to the command line parser.
This commit fixes the issue by moving the parsing code until after the arguments are registered.
Solves scylladb#4141.

Signed-off-by: Kamil Braun <kbraun@scylladb.com>
kbr- added a commit to kbr-/scylla that referenced this issue Jul 2, 2019
Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run.
The first parse is there to check if the "--version" flag is present --- in this case the version is printed
and the program exists.  The second parsing is correct; however, most of the arguments were improperly treated
as positional arguments during the first parsing (e.g., "--network host" would treat "host" as a positional argument).
This happened because the arguments weren't known to the command line parser.
This commit fixes the issue by moving the parsing code until after the arguments are registered.
Resolves scylladb#4141.

Signed-off-by: Kamil Braun <kbraun@scylladb.com>
@slivne slivne added this to the 3.2 milestone Jul 8, 2019
avikivity added a commit that referenced this issue Jul 10, 2019
"
Fixes #4203 and #4141.
"

* 'cmdline' of https://github.com/kbr-/scylla:
  Add logging of parsed command line options
  Fix command line argument parsing in main.
avikivity added a commit that referenced this issue Aug 13, 2019
Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run.
The first parse is there to check if the "--version" flag is present --- in this case the version is printed
and the program exists.  The second parsing is correct; however, most of the arguments were improperly treated
as positional arguments during the first parsing (e.g., "--network host" would treat "host" as a positional argument).
This happened because the arguments weren't known to the command line parser.
This commit fixes the issue by moving the parsing code until after the arguments are registered.
Resolves #4141.

Signed-off-by: Kamil Braun <kbraun@scylladb.com>
(cherry picked from commit f155a2d)
avikivity added a commit that referenced this issue Aug 13, 2019
Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run.
The first parse is there to check if the "--version" flag is present --- in this case the version is printed
and the program exists.  The second parsing is correct; however, most of the arguments were improperly treated
as positional arguments during the first parsing (e.g., "--network host" would treat "host" as a positional argument).
This happened because the arguments weren't known to the command line parser.
This commit fixes the issue by moving the parsing code until after the arguments are registered.
Resolves #4141.

Signed-off-by: Kamil Braun <kbraun@scylladb.com>
(cherry picked from commit f155a2d)
@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Aug 13, 2019

Backported to 3.1, 3.0.

@tzach

This comment has been minimized.

Copy link
Contributor

@tzach tzach commented Aug 22, 2019

@CountZukula please check with latest Scylla Docker 3.0.10

@CountZukula

This comment has been minimized.

Copy link
Author

@CountZukula CountZukula commented Aug 22, 2019

@tzach It's been a while but I tried to reproduce my environment and ran the docker command as stated in the first post. Getting the same result, full output:

jnieland@node0:~$ sudo docker run --name scylla -it --network host scylladb/scylla        
Unable to find image 'scylladb/scylla:latest' locally
latest: Pulling from scylladb/scylla
8ba884070f61: Pull complete 
cd4f8f8c60fc: Pull complete 
2747a5fb8f41: Pull complete 
07583ab71a18: Pull complete 
5fcac9cdadf6: Pull complete 
c690c84c7597: Pull complete 
63ea31381ef0: Pull complete 
551655fd09ec: Pull complete 
a7efd0f525b1: Pull complete 
ba3549fdb516: Pull complete 
a6c1be1d6b52: Pull complete 
76fef7b03810: Pull complete 
26114236ac85: Pull complete 
402cb8658fe9: Pull complete 
Digest: sha256:e7f861e62f363f9080af9369ef2831039d8aeb1d6a8c3d463824831762d37f26
Status: Downloaded newer image for scylladb/scylla:latest
running: (['/usr/lib/scylla/scylla_dev_mode_setup', '--developer-mode', '1'],)
running: (['/usr/lib/scylla/scylla_io_setup'],)
2019-08-22 13:40:30,578 CRIT Supervisor running as root (no user in config file)
2019-08-22 13:40:30,578 WARN Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing
2019-08-22 13:40:30,578 WARN Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing
2019-08-22 13:40:30,578 WARN Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing
2019-08-22 13:40:30,593 INFO RPC interface 'supervisor' initialized
2019-08-22 13:40:30,593 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2019-08-22 13:40:30,593 INFO supervisord started with pid 10
2019-08-22 13:40:31,597 INFO spawned: 'scylla-jmx' with pid 13
2019-08-22 13:40:31,599 INFO spawned: 'scylla' with pid 14
2019-08-22 13:40:31,602 INFO spawned: 'scylla-housekeeping' with pid 15
error: too many positional options have been specified on the command line

Try --help.
2019-08-22 13:40:31,713 INFO exited: scylla (exit status 2; not expected)
Connecting to http://localhost:10000
Starting the JMX server
JMX is enabled to receive remote connections on port: 7199
2019-08-22 13:40:33,063 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-08-22 13:40:33,065 INFO spawned: 'scylla' with pid 32
2019-08-22 13:40:33,065 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
error: too many positional options have been specified on the command line

Try --help.
2019-08-22 13:40:33,174 INFO exited: scylla (exit status 2; not expected)
2019-08-22 13:40:35,179 INFO spawned: 'scylla' with pid 34
error: too many positional options have been specified on the command line

Try --help.
2019-08-22 13:40:35,287 INFO exited: scylla (exit status 2; not expected)
Traceback (most recent call last):
  File "/usr/lib/scylla/scylla-housekeeping", line 174, in <module>
    args.func(args)
  File "/usr/lib/scylla/scylla-housekeeping", line 101, in check_version
    current_version = sanitize_version(get_api('/storage_service/scylla_release_version'))
  File "/usr/lib/scylla/scylla-housekeeping", line 65, in get_api
    return get_json_from_url("http://" + api_address + path)
  File "/usr/lib/scylla/scylla-housekeeping", line 62, in get_json_from_url
    return json.loads(data.decode('utf-8'))
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
2019-08-22 13:40:38,711 INFO spawned: 'scylla' with pid 44
error: too many positional options have been specified on the command line

Try --help.
2019-08-22 13:40:38,829 INFO exited: scylla (exit status 2; not expected)
2019-08-22 13:40:39,830 INFO gave up: scylla entered FATAL state, too many start retries too quickly
^C2019-08-22 13:43:12,461 WARN received SIGINT indicating exit request
2019-08-22 13:43:12,461 INFO waiting for scylla-jmx, scylla-housekeeping to die
2019-08-22 13:43:12,462 INFO stopped: scylla-housekeeping (terminated by SIGTERM)

@CountZukula

This comment has been minimized.

Copy link
Author

@CountZukula CountZukula commented Aug 22, 2019

(for reference, LATEST in the above was 3.0.10)

@tzach tzach reopened this Aug 22, 2019
@tzach

This comment has been minimized.

Copy link
Contributor

@tzach tzach commented Aug 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.