Docker 'network=host' doesn't start - too many positional options have been specified #4141

joachimnielandt · 2019-01-24T13:37:08Z

This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.

I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): Docker version 3.0.1 / LATEST
Cluster size: Not relevant
OS (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu (Docker)

Hardware details (for performance issues) Delete if unneeded
Platform (physical/VM/cloud instance type/docker): docker, running on physical node managed on cluster
Hardware: sockets= cores=4 hyperthreading=no memory=16G
Disks: (SSD/HDD, count) SSD, 16GB

I am trying to run Scylla on multiple virtual nodes, on which I have administrator access. The nodes can ping each other and are available on the same network. Currently, I'm trying to get it up and running using Docker, as I have tried beforehand on my local machine (which seemed to be going fine).

Now, on one of the virtual nodes, I am executing the following (simplified command as this already triggers my problem):

sudo docker run --name scylla -it --network host scylladb/scylla

I'm getting the following error, resulting in Scylla not starting up (error: too many positional options have been specified on the command line):

running: (['/usr/lib/scylla/scylla_dev_mode_setup', '--developer-mode', '1'],)
running: (['/usr/lib/scylla/scylla_io_setup'],)
2019-01-24 13:31:09,884 CRIT Supervisor running as root (no user in config file)
2019-01-24 13:31:09,884 WARN Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing
2019-01-24 13:31:09,884 WARN Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing
2019-01-24 13:31:09,884 WARN Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing
2019-01-24 13:31:09,899 INFO RPC interface 'supervisor' initialized
2019-01-24 13:31:09,899 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2019-01-24 13:31:09,899 INFO supervisord started with pid 10
2019-01-24 13:31:10,901 INFO spawned: 'scylla-jmx' with pid 13
2019-01-24 13:31:10,904 INFO spawned: 'scylla' with pid 14
2019-01-24 13:31:10,906 INFO spawned: 'scylla-housekeeping' with pid 15
error: too many positional options have been specified on the command line

Try --help.
2019-01-24 13:31:11,018 INFO exited: scylla (exit status 2; not expected)
Connecting to http://localhost:10000
Starting the JMX server
JMX is enabled to receive remote connections on port: 7199
2019-01-24 13:31:12,369 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-01-24 13:31:12,371 INFO spawned: 'scylla' with pid 32
2019-01-24 13:31:12,371 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
error: too many positional options have been specified on the command line

Try --help.
2019-01-24 13:31:12,480 INFO exited: scylla (exit status 2; not expected)
2019-01-24 13:31:14,485 INFO spawned: 'scylla' with pid 34
error: too many positional options have been specified on the command line

Try --help.

When removing 'network host' from the docker command, however, Scylla starts as expected.

The text was updated successfully, but these errors were encountered:

slivne · 2019-01-27T11:32:45Z

checked locally on my machine sudo docker run --name scylla -it --network host scylladb/scylla

works
latest is 3.0.1

maybe related to running on ubuntu - although its not clear how that can cause an issue.

slivne · 2019-02-07T11:00:41Z

#4203 would help to solve such issues

tzemanovic · 2019-02-21T10:55:59Z

I'm hitting the same issue running Docker on mac

Fixes scylladb#4203 and scylladb#4141. Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run. The first parse is there to check if the "--version" flag is present --- in this case the version is printed and the program exists. However, most of the arguments were improperly treated as positional arguments the first time they were parsed (e.g., "--network host" would treat "host" as a positional argument). This commit fixes the issue and prints the parsed arguments. Signed-off-by: Kamil Braun <kbraun@scylladb.com>

Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run. The first parse is there to check if the "--version" flag is present --- in this case the version is printed and the program exists. The second parsing is correct; however, most of the arguments were improperly treated as positional arguments during the first parsing (e.g., "--network host" would treat "host" as a positional argument). This happened because the arguments weren't known to the command line parser. This commit fixes the issue by moving the parsing code until after the arguments are registered. Solves scylladb#4141. Signed-off-by: Kamil Braun <kbraun@scylladb.com>

Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run. The first parse is there to check if the "--version" flag is present --- in this case the version is printed and the program exists. The second parsing is correct; however, most of the arguments were improperly treated as positional arguments during the first parsing (e.g., "--network host" would treat "host" as a positional argument). This happened because the arguments weren't known to the command line parser. This commit fixes the issue by moving the parsing code until after the arguments are registered. Resolves scylladb#4141. Signed-off-by: Kamil Braun <kbraun@scylladb.com>

" Fixes #4203 and #4141. " * 'cmdline' of https://github.com/kbr-/scylla: Add logging of parsed command line options Fix command line argument parsing in main.

Command line arguments are parsed twice in Scylla: once in main and once in Seastar's app_template::run. The first parse is there to check if the "--version" flag is present --- in this case the version is printed and the program exists. The second parsing is correct; however, most of the arguments were improperly treated as positional arguments during the first parsing (e.g., "--network host" would treat "host" as a positional argument). This happened because the arguments weren't known to the command line parser. This commit fixes the issue by moving the parsing code until after the arguments are registered. Resolves #4141. Signed-off-by: Kamil Braun <kbraun@scylladb.com> (cherry picked from commit f155a2d)

avikivity · 2019-08-13T17:13:53Z

Backported to 3.1, 3.0.

tzach · 2019-08-22T09:43:29Z

@CountZukula please check with latest Scylla Docker 3.0.10

joachimnielandt · 2019-08-22T13:45:38Z

@tzach It's been a while but I tried to reproduce my environment and ran the docker command as stated in the first post. Getting the same result, full output:

jnieland@node0:~$ sudo docker run --name scylla -it --network host scylladb/scylla        
Unable to find image 'scylladb/scylla:latest' locally
latest: Pulling from scylladb/scylla
8ba884070f61: Pull complete 
cd4f8f8c60fc: Pull complete 
2747a5fb8f41: Pull complete 
07583ab71a18: Pull complete 
5fcac9cdadf6: Pull complete 
c690c84c7597: Pull complete 
63ea31381ef0: Pull complete 
551655fd09ec: Pull complete 
a7efd0f525b1: Pull complete 
ba3549fdb516: Pull complete 
a6c1be1d6b52: Pull complete 
76fef7b03810: Pull complete 
26114236ac85: Pull complete 
402cb8658fe9: Pull complete 
Digest: sha256:e7f861e62f363f9080af9369ef2831039d8aeb1d6a8c3d463824831762d37f26
Status: Downloaded newer image for scylladb/scylla:latest
running: (['/usr/lib/scylla/scylla_dev_mode_setup', '--developer-mode', '1'],)
running: (['/usr/lib/scylla/scylla_io_setup'],)
2019-08-22 13:40:30,578 CRIT Supervisor running as root (no user in config file)
2019-08-22 13:40:30,578 WARN Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing
2019-08-22 13:40:30,578 WARN Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing
2019-08-22 13:40:30,578 WARN Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing
2019-08-22 13:40:30,593 INFO RPC interface 'supervisor' initialized
2019-08-22 13:40:30,593 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2019-08-22 13:40:30,593 INFO supervisord started with pid 10
2019-08-22 13:40:31,597 INFO spawned: 'scylla-jmx' with pid 13
2019-08-22 13:40:31,599 INFO spawned: 'scylla' with pid 14
2019-08-22 13:40:31,602 INFO spawned: 'scylla-housekeeping' with pid 15
error: too many positional options have been specified on the command line

Try --help.
2019-08-22 13:40:31,713 INFO exited: scylla (exit status 2; not expected)
Connecting to http://localhost:10000
Starting the JMX server
JMX is enabled to receive remote connections on port: 7199
2019-08-22 13:40:33,063 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-08-22 13:40:33,065 INFO spawned: 'scylla' with pid 32
2019-08-22 13:40:33,065 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
error: too many positional options have been specified on the command line

Try --help.
2019-08-22 13:40:33,174 INFO exited: scylla (exit status 2; not expected)
2019-08-22 13:40:35,179 INFO spawned: 'scylla' with pid 34
error: too many positional options have been specified on the command line

Try --help.
2019-08-22 13:40:35,287 INFO exited: scylla (exit status 2; not expected)
Traceback (most recent call last):
  File "/usr/lib/scylla/scylla-housekeeping", line 174, in <module>
    args.func(args)
  File "/usr/lib/scylla/scylla-housekeeping", line 101, in check_version
    current_version = sanitize_version(get_api('/storage_service/scylla_release_version'))
  File "/usr/lib/scylla/scylla-housekeeping", line 65, in get_api
    return get_json_from_url("http://" + api_address + path)
  File "/usr/lib/scylla/scylla-housekeeping", line 62, in get_json_from_url
    return json.loads(data.decode('utf-8'))
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
2019-08-22 13:40:38,711 INFO spawned: 'scylla' with pid 44
error: too many positional options have been specified on the command line

Try --help.
2019-08-22 13:40:38,829 INFO exited: scylla (exit status 2; not expected)
2019-08-22 13:40:39,830 INFO gave up: scylla entered FATAL state, too many start retries too quickly
^C2019-08-22 13:43:12,461 WARN received SIGINT indicating exit request
2019-08-22 13:43:12,461 INFO waiting for scylla-jmx, scylla-housekeeping to die
2019-08-22 13:43:12,462 INFO stopped: scylla-housekeeping (terminated by SIGTERM)

joachimnielandt · 2019-08-22T13:47:40Z

(for reference, LATEST in the above was 3.0.10)

tzach · 2019-08-22T14:09:45Z

@penberg ?

kayvan-eth · 2019-12-23T07:24:01Z

when you set network mode to host, scylla docker's entrypoint will look for network interfaces to set the listen address, but when it will reach too many options to set as listen address, so for running container with network mode host, just add listen address argument to cmd.

docker run --name scylla -it --network host scylladb/scylla --listen-address $HOST_IP

elbakerino · 2021-07-31T18:58:25Z

Is also caused when using multiple networks in docker-compose.yml (either using docker compose or swarm).

Solution for me was: using only one network - then scylla sorted it out by itself.

Initially used one network for the app services and one which spans only monitored services, thus with only the app-net I've needed to publish the prometheus port to be able to access it directly.

nyh · 2021-08-26T07:52:48Z

Issue #9240 seems to have encountered this bug again. He was trying to run Alternator but it doesn't seem that there is anything in "too many positional options" specific to Alternator - rather the problem appears to be that our startup script passes to Scylla both IPv4 and IPv6 addresses with a space between them, where just a single IP address is expected.

nyh · 2021-08-29T07:23:40Z

An explanation what the user in #9240 was trying to do when encountering this bug: HENNGE/aiodynamo#27 (comment)

Horcrux7 · 2022-08-29T09:33:54Z

Is there an workaround to handle this in a dual stacked docker test system?

joachimnielandt changed the title ~~Docker 'network=host'~~ Docker 'network=host' doesn't start - too many positional options have been specified Jan 24, 2019

slivne added the area/docker label Jan 27, 2019

slivne assigned penberg Jan 27, 2019

kbr- mentioned this issue Jul 1, 2019

Fix command line parsing and add logging. #4631

Merged

slivne added this to the 3.2 milestone Jul 8, 2019

avikivity added a commit that referenced this issue Jul 10, 2019

Merge "Fix command line parsing and add logging." from Kamil

138b28a

" Fixes #4203 and #4141. " * 'cmdline' of https://github.com/kbr-/scylla: Add logging of parsed command line options Fix command line argument parsing in main.

scylladb-promoter added the Backport candidate label Jul 11, 2019

scylladb-promoter closed this as completed in f155a2d Jul 11, 2019

avikivity removed the Backport candidate label Aug 13, 2019

tzach reopened this Aug 22, 2019

slivne modified the milestones: 3.2, 3.4 Mar 5, 2020

slivne modified the milestones: 4.0, 4.1 Mar 24, 2020

slivne removed this from the 4.1 milestone Jun 1, 2020

slivne added this to the 4.2 milestone Jun 1, 2020

slivne modified the milestones: 4.2, 4.3 Nov 26, 2020

slivne added the onboarding label Mar 29, 2021

slivne modified the milestones: 4.3, 4.5 Mar 29, 2021

dimaqq mentioned this issue Aug 26, 2021

error: too many positional options have been specified on the command line when trying alternator #9240

Closed

1 task

penberg removed their assignment Sep 19, 2021

slivne modified the milestones: 4.5, 4.6 Nov 10, 2021

slivne modified the milestones: 4.6, 5.0 Feb 8, 2022

DoronArazii modified the milestones: 5.0, 5.x Oct 12, 2022

dani-tweig added good first issue and removed onboarding labels Feb 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker 'network=host' doesn't start - too many positional options have been specified #4141

Docker 'network=host' doesn't start - too many positional options have been specified #4141

joachimnielandt commented Jan 24, 2019 •

edited

slivne commented Jan 27, 2019

slivne commented Feb 7, 2019

tzemanovic commented Feb 21, 2019

avikivity commented Aug 13, 2019

tzach commented Aug 22, 2019 •

edited

joachimnielandt commented Aug 22, 2019

joachimnielandt commented Aug 22, 2019

tzach commented Aug 22, 2019

kayvan-eth commented Dec 23, 2019

elbakerino commented Jul 31, 2021

nyh commented Aug 26, 2021

nyh commented Aug 29, 2021

Horcrux7 commented Aug 29, 2022

Docker 'network=host' doesn't start - too many positional options have been specified #4141

Docker 'network=host' doesn't start - too many positional options have been specified #4141

Comments

joachimnielandt commented Jan 24, 2019 • edited

slivne commented Jan 27, 2019

slivne commented Feb 7, 2019

tzemanovic commented Feb 21, 2019

avikivity commented Aug 13, 2019

tzach commented Aug 22, 2019 • edited

joachimnielandt commented Aug 22, 2019

joachimnielandt commented Aug 22, 2019

tzach commented Aug 22, 2019

kayvan-eth commented Dec 23, 2019

elbakerino commented Jul 31, 2021

nyh commented Aug 26, 2021

nyh commented Aug 29, 2021

Horcrux7 commented Aug 29, 2022

joachimnielandt commented Jan 24, 2019 •

edited

tzach commented Aug 22, 2019 •

edited