[BUG] Error unmarshaling return; nested exception is: java.lang.ClassNotFoundException: org.apache.cassandra.io.FSWriteError (no security manager: RMI class loader disabled)'}]

[bmonteiro]

Describe the bug
One node out of X with similar setup is throwing this error:

[pool-4-thread-1] ERROR com.instaclustr.operations.Operation - Operation 247f4ef6-ee93-43cf-ad8f-199423196d8c has failed.
[ResultEntry{failed=false, operation='BackupOperation{id=247f4ef6-ee93-43cf-ad8f-199423196d8c, creationTime=2020-10-16T18:47:32.533Z, request=BackupOperationRequest{storageLocation=StorageLocation{rawLocation=s3://xxxx, storageProvider=s3, bucket=xxxx, clusterId=xxxxx, datacenterId=xxxx, nodeId=xxxx, fileBackupDirectory=null, cloudLocation=true}, duration=null, bandwidth=null, concurrentConnections=10, cassandraDirectory=/data/cassandra, entities=DatabaseEntities{keyspaces=[xxxxx], keyspacesAndTables={}}, snapshotTag=cassandra-dev-2020-10-16-839210cc-6834-3c3d-afed-a4052e6d54b7-1602874052628, k8sNamespace=default, k8sSecretName=null, globalRequest=false, dc=null, timeout=5, metadataDirective=COPY, insecure=false, schemaVersion=839210cc-6834-3c3d-afed-a4052e6d54b7}, state=RUNNING, failureCause=null, progress=0.0, startTime=2020-10-16T18:47:32.542Z, shouldCancel=false}', exceptionMessage='java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:
java.lang.ClassNotFoundException: org.apache.cassandra.io.FSWriteError (no security manager: RMI class loader disabled)'}]
at com.instaclustr.cassandra.backup.impl.backup.BackupOperation.run0(BackupOperation.java:113)
at com.instaclustr.operations.Operation.run(Operation.java:106)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
java.lang.IllegalStateException: Backup operation was not successful:
at com.instaclustr.cassandra.backup.cli.BackupApplication.run(BackupApplication.java:65)
at picocli.CommandLine.executeUserObject(CommandLine.java:1729)
at picocli.CommandLine.access$900(CommandLine.java:145)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2101)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2068)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
at picocli.CommandLine.execute(CommandLine.java:1864)
at com.instaclustr.picocli.CLIApplication.execute(CLIApplication.java:34)
at com.instaclustr.cassandra.backup.cli.BackupRestoreCLI.main(BackupRestoreCLI.java:63)
at com.instaclustr.cassandra.backup.cli.BackupRestoreCLI.main(BackupRestoreCLI.java:55)

After running the tool:
/usr/bin/java -jar instaclustr-backup-restore.jar backup --jmx-service 127.0.0.1:7199 --storage-location=s3://yyyy/xxx/dc-bbbb/nodeX --data-directory=/data/cassandra --entities=xxxx --snapshot-tag=cassandra-dev-date +%F

To Reproduce
/usr/bin/java -jar instaclustr-backup-restore.jar backup --jmx-service 127.0.0.1:7199 --storage-location=s3://yyyy/xxx/dc-bbbb/nodeX --data-directory=/data/cassandra --entities=xxxx --snapshot-tag=cassandra-dev-date +%F

Expected behavior
...

System and versions (please complete the following information):
...

Additional context
...

CC: @smiklosovic

[smiklosovic]

Hi @bmonteiro

this is the cause:

exceptionMessage='java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:
java.lang.ClassNotFoundException: org.apache.cassandra.io.FSWriteError (no security manager: RMI class loader disabled)'}]

This is very strange error. It seems like FSWriteError is not on a classpath but I am thinking hard about why ... What is the version of Cassandra you are using?

[smiklosovic]

Please keep in mind that Esop is talking to Cassandra only via JMX.

This is interesting too: no security manager: RMI class loader disabled

I think it has something to do with this https://stackoverflow.com/questions/6322107/java-no-security-manager-rmi-class-loader-disabled#6348219

Maybe the reason is that the Java interfaces I am using for JMX are in a different package than what they trully are in but the fact that there is ClassNotFoundException is puzzling ...

[bmonteiro]

Hi @bmonteiro

this is the cause:

exceptionMessage='java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:
java.lang.ClassNotFoundException: org.apache.cassandra.io.FSWriteError (no security manager: RMI class loader disabled)'}]

This is very strange error. It seems like FSWriteError is not on a classpath but I am thinking hard about why ... What is the version of Cassandra you are using?

@smiklosovic ,
cassandra:
Installed: 3.11.3
Candidate: 3.11.3
Version table:
*** 3.11.3 100
100 /var/lib/dpkg/status

[timprijn]

Hi,

Last week I ran into the same exception when testing Icarus in a Docker container. In our case this was related to file permissions. Our Cassandra instance ran under a different user (non-root) than our backup container. After correcting this it was resolved.

Perhaps this answer helps in your situation as well.

[smiklosovic]

@bmonteiro would you please try the suggested solution above, please?

[bmonteiro]

@timprijn @smiklosovic , thx, indeed there was a folder with owner root and all others with cassandra user once updated with "chown -R cassandra:cassandra /data/cassandra/" all is working fine. Thx guys!