problem deploying cassandra-mesos with marathon

[kikicarbonell]

I'm trying cassandra-mesos on my private cluster os mesos. I'm using Readme instruction to deploy with marathon but some error occure on wake up task:

Stderr output:

I0925 15:55:06.185039  8383 fetcher.cpp:214] Fetching URI 'https://downloads.mesosphere.io/cassandra-mesos/artifacts/0.2.1-SNAPSHOT-589-master-4c6502b0a6/cassandra-mesos-0.2.1-SNAPSHOT-589-master-4c6502b0a6.tar.gz'
I0925 15:55:06.185165  8383 fetcher.cpp:125] Fetching URI 'https://downloads.mesosphere.io/cassandra-mesos/artifacts/0.2.1-SNAPSHOT-589-master-4c6502b0a6/cassandra-mesos-0.2.1-SNAPSHOT-589-master-4c6502b0a6.tar.gz' with os::net
I0925 15:55:06.185180  8383 fetcher.cpp:135] Downloading 'https://downloads.mesosphere.io/cassandra-mesos/artifacts/0.2.1-SNAPSHOT-589-master-4c6502b0a6/cassandra-mesos-0.2.1-SNAPSHOT-589-master-4c6502b0a6.tar.gz' to '/tmp/mesos/slaves/20150925-135709-503717292-5050-2136-S0/frameworks/20150925-144548-503717292-5050-5066-0001/executors/cassandra-dev-test.51620664-63bf-11e5-b350-56847afe9799/runs/e0aaadd9-9c86-47e1-8df8-d2c437f960f4/cassandra-mesos-0.2.0-1.tar.gz'
I0925 15:56:07.968350  8383 fetcher.cpp:78] Extracted resource '/tmp/mesos/slaves/20150925-135709-503717292-5050-2136-S0/frameworks/20150925-144548-503717292-5050-5066-0001/executors/cassandra-dev-test.51620664-63bf-11e5-b350-56847afe9799/runs/e0aaadd9-9c86-47e1-8df8-d2c437f960f4/cassandra-mesos-0.2.0-1.tar.gz' into '/tmp/mesos/slaves/20150925-135709-503717292-5050-2136-S0/frameworks/20150925-144548-503717292-5050-5066-0001/executors/cassandra-dev-test.51620664-63bf-11e5-b350-56847afe9799/runs/e0aaadd9-9c86-47e1-8df8-d2c437f960f4'
I0925 15:56:07.971684  8383 fetcher.cpp:214] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz'
I0925 15:56:07.971709  8383 fetcher.cpp:125] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' with os::net
I0925 15:56:07.971725  8383 fetcher.cpp:135] Downloading 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' to '/tmp/mesos/slaves/20150925-135709-503717292-5050-2136-S0/frameworks/20150925-144548-503717292-5050-5066-0001/executors/cassandra-dev-test.51620664-63bf-11e5-b350-56847afe9799/runs/e0aaadd9-9c86-47e1-8df8-d2c437f960f4/jre-7u76-linux-x64.tar.gz'
I0925 15:56:51.630692  8383 fetcher.cpp:78] Extracted resource '/tmp/mesos/slaves/20150925-135709-503717292-5050-2136-S0/frameworks/20150925-144548-503717292-5050-5066-0001/executors/cassandra-dev-test.51620664-63bf-11e5-b350-56847afe9799/runs/e0aaadd9-9c86-47e1-8df8-d2c437f960f4/jre-7u76-linux-x64.tar.gz' into '/tmp/mesos/slaves/20150925-135709-503717292-5050-2136-S0/frameworks/20150925-144548-503717292-5050-5066-0001/executors/cassandra-dev-test.51620664-63bf-11e5-b350-56847afe9799/runs/e0aaadd9-9c86-47e1-8df8-d2c437f960f4'
I0925 15:56:51.720883  8381 exec.cpp:132] Version: 0.22.1
I0925 15:56:51.723655  8426 exec.cpp:206] Executor registered on slave 20150925-135709-503717292-5050-2136-S0
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@log_env@716: Client environment:host.name=vcmms.domain.com
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@log_env@724: Client environment:os.arch=3.19.3-1.el6.x86_64
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Mon Mar 30 13:50:16 EDT 2015
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/mesos/slaves/20150925-135709-503717292-5050-2136-S0/frameworks/20150925-144548-503717292-5050-5066-0001/executors/cassandra-dev-test.51620664-63bf-11e5-b350-56847afe9799/runs/e0aaadd9-9c86-47e1-8df8-d2c437f960f4
2015-09-25 15:56:52,376:8432(0x7f3cb2ffd700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=172.29.6.30:2181 sessionTimeout=10000 watcher=0x7f3cee4ceeb0 sessionId=0 sessionPasswd=<null> context=0x7f3c9c000930 flags=0
2015-09-25 15:56:52,377:8432(0x7f3cb17fa700):ZOO_INFO@check_events@1703: initiated connection to server [172.29.6.30:2181]
2015-09-25 15:56:52,382:8432(0x7f3cb17fa700):ZOO_INFO@check_events@1750: session establishment complete on server [172.29.6.30:2181], sessionId=0x15005d328430014, negotiated timeout=10000

and Stdout output:

Registered executor on vcmms.domain.com
Starting task cassandra-dev-test.51620664-63bf-11e5-b350-56847afe9799
Forked command at 8432
sh -c '$(pwd)/jre*/bin/java $JAVA_OPTS -classpath cassandra-mesos-framework.jar io.mesosphere.mesos.frameworks.cassandra.framework.Main'
Command exited with status 10 (pid: 8432)

My cluster has:
mesos masters: 1
mesos slaves: 3
marathon
docker

[BenWhitehead]

@kikicarbonell Can you also attache cassandra-mesos.log from the sandbox that contains the actual logs from the scheduler process.

[kikicarbonell]

@BenWhitehead please, can you read all logs from: http://stackoverflow.com/questions/32830901/cassandra-mesos-health-check-fail-when-deploy-with-marathon

[BenWhitehead]

Hi @kikicarbonell The ports used by cassandra can be found here in the official documentation. We don't start OpsCenter so you can disregard those ports.

[kikicarbonell]

@BenWhitehead well I was opening port: 7000,7001,9042,9160,7199,22 but not work? if possible that made some problem around mesos framework?

[BenWhitehead]

Possibly, I thought you were saying that the farmwork tried to start the tasks but cassandra wasn't able to ever form the ring.

I'll try and outline all of the network connections I know about in play here.

1. Cassandra-Mesos framework scheduler need to communicate with zk 2181 to persist it's state, locate the leading mesos-master. Needs to be able to communicate with the mesos master at port 5050.
2. The mesos-master then needs to be able to communicate back with the scheduler on a random port selected by libmesos at runtime (it's in the random port range the kernel reserves).
3. marathon will need to be able to communicate with the scheduler on $PORT0 in order to issue health checks against the http api.
4. The Custom Executor needs to be able to connect to 5051 to communicate with the mesos-agent
5. The mesos-agent will need to be able to communicate back with the executor on a random port selected by libmesos at runtime (it's in the random port range the kernel reserves).
6. Each Server task will bind against 7000,7001,7199,9042,9160. Port 7000 is used by cassandra for it's internode gossip communication.
7. Each Custom Executor will need to be able to communicate with localhost:7199 in order to connect to the JMX MBeans for nodetool commands (health checks, cluster jobs, shutdown, etc.)
8. Each Custom Executor will need to be able to communicate with the random scheduler port bound in note 2 above. (this is for Framework Messages sent from the custom executor to the scheduler with information about health check and a couple other things that don't necessitate a task.)
9. Then each client will need to be able to access 9042/9160 of each node running a server so that it can interact with cassandra itself.

Hopefully this can help you track down the packet breakdown. I'd be interested in knowing what ends up working for you.

[kikicarbonell]

@BenWhitehead thks for your detailed explanation about process and port used on
cassandra-mesos fremework, after this I suppouse that the problem will be
the random port used by mesos (step 5 in your description), then I create a
rule on iptable to allow tcp traffic betwen al mesos nodes, it' s a
temporal solution but it work.

The iptable command to do it:

iptable -A INPUT -p tcp -s <hostname> -j ACCEPT;

[triclambert]

This repository is deprecated. Please use the latest supported framework here: https://github.com/mesosphere/dcos-cassandra-service, and provide feedback for us here: https://dcosjira.atlassian.net/browse/CASSANDRA.