question: backup cassandra docker

[vbpatil449]

Does this tool support backing up data from Cassandra docker container?

I am getting this error -
com.google.inject.CreationException: Unable to create injector, see the following errors:

1. Checked provider threw exception.
   while locating com.instaclustr.kubernetes.KubernetesApiModule$ClientCoreV1ApiProvider
   at com.instaclustr.kubernetes.KubernetesApiModule.configure(KubernetesApiModule.java:23) (via modules: com.instaclustr.esop.guice.StorageModules -> com.instaclustr.kubernetes.KubernetesApiModule)
   while locating io.kubernetes.client.apis.CoreV1Api

1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
at com.google.inject.Guice.createInjector(Guice.java:87)
at com.instaclustr.esop.cli.Esop.init(Esop.java:102)
at com.instaclustr.esop.cli.BackupApplication.run(BackupApplication.java:63)

[smiklosovic]

hi @vbpatil449 , yes it should and if it does not it is wrong, do you have just this log? I think there should be complete stracktrace with the cause.

It fails somewhere here

                    ApiClient client = ClientBuilder.defaultClient();
                    Configuration.setDefaultApiClient(client);

I think your Docker does not know anything about Kubernetes so it fails or similar ... getting more logs would definitely help. I will check this for you no worries.

[vbpatil449]

Thanks for getting back. I think the issue is because docker is on a different network(docker-network) - host doesn't have access to jmxrmi port I think.

I was able to run the the backup inside the docker network (containers) and as well as backup-commitlog

I am using
--storage-location=file:////var/lib/cassandra/backme
--data-directory=/var/lib/cassandra
--snapshot-tag=snappy123

Can you provide the commands to restore the snapshot and as well as restore commit-log ?

I tried hardlinks steps as per the README - which seems to fail in my case due to local File ...

22:59:19.269 ERROR com.instaclustr.esop.impl.restore.RestorationPhase$HardlinkingPhase - Unable to create a hardlink from /tmp/download/sstables/test/emp-d0ef1b806b3211eba5d19116fc548b6b/md-1-big-CompressionInfo.db to /var/lib/cassandra/data/test/emp-d0ef1b806b3211eba5d19116fc548b6b/md-1-big-CompressionInfo.db, skipping the linking of all other resources and deleting already linked ones.
java.nio.file.FileSystemException: /var/lib/cassandra/data/test/emp-d0ef1b806b3211eba5d19116fc548b6b/md-1-big-CompressionInfo.db -> /tmp/download/sstables/test/emp-d0ef1b806b3211eba5d19116fc548b6b/md-1-big-CompressionInfo.db: Invalid cross-device link

[smiklosovic]

Hi @vbpatil449 ,

Do I understand it right that your original exception has disappeared and it is not a problem anymore as you have successfully backed up your data from inside of a Docker container?

Yes, your restore failed because you are trying to hard-link a file from outside of a container to a file inside of a container and that is not possible in Docker. You may link only files which are inside container so I think that your restoration has to be run in Docker container and files will be stored in container too hence linking will not fail.

Examples of commit log and normal restoration might be found here:

https://github.com/instaclustr/esop#backup-and-restore-of-commit-logs
https://github.com/instaclustr/esop#examples-of-command-line-invocation

You will find all information there just by reading it.

[vbpatil449]

thanks for the details..
yes I am actually running these commands inside cassandra docker. I see that the volume (/var/lib/cassandra/data) is external volume mapped to the container.

For restore - As the backup files are accessible locally, do i have to use hardlinks? why do i have to specify download? is there a different option to just copy snapshots files ?

Also to restore - Should the cassandra service be running or stopped ?

[smiklosovic]

Yes you have to use hard links, download will download it to some dir different from cassandra dir and then it will hardlink it to target cassandra dir. Sure you have to specify download if you want to download them. Please go once again over examples, there is example of hardlinking importing, 4 commands, download, truncate, import, cleanup - this is in case you have Cassandra running. In case it is not running, you have to use in-place strategy. If you want to restore on a running cluster, you have to do it in connection with Icarus (our other project), Esop can restore just that one Cassandra node, not whole cluster, but you can still choose if you want that to restore on a running node or on stopped.

[vbpatil449]

Thank you.. I am following the same steps to restore...still get this error

14:36:43.609 ERROR com.instaclustr.operations.Operation - Reporting errors for operation 002ac5c6-3dde-4dfc-9aa1-54ce61f9e986
14:36:43.610 ERROR com.instaclustr.operations.Operation - --- start of error reporting ---
14:36:43.611 ERROR com.instaclustr.operations.Operation - Error{source=cassandra-0-seed, message=Some keyspaces to process are not present in Cassandra: [test], throwable=java.lang.IllegalStateException: Unable to initialise phase DOWNLOAD: }
java.lang.IllegalStateException: Unable to initialise phase DOWNLOAD:

I do have 'test' keyspace created

here is my restore script looks
`echo " The first restoration phase is DOWNLOAD as we need to download remote SSTables "
echo " "
"$JAVA" -jar "./esop.jar"
restore
--storage-location=file:////var/lib/cassandra/backme
--data-directory=/var/lib/cassandra/data
--entities=test
--snapshot-tag=snappy123
--restoration-strategy-type=hardlinks
--restoration-phase-type=download
--import-source-dir=/var/lib/cassandra/download/sstables

echo " Then we need to truncate keyspace"
echo " "
"$JAVA" -jar "./esop.jar"
restore
--storage-location=file:////var/lib/cassandra/backme
--data-directory=/var/lib/cassandra/data
--entities=test
--snapshot-tag=snappy123
--restoration-strategy-type=hardlinks
--restoration-phase-type=truncate
--import-source-dir=/var/lib/cassandra/download/sstables

echo " we can make hardlinks from directory"
echo " "
"$JAVA" -jar "./esop.jar"
restore
--storage-location=file:////var/lib/cassandra/backme
--data-directory=/var/lib/cassandra/data
--entities=test
--snapshot-tag=snappy123
--restoration-strategy-type=hardlinks
--restoration-phase-type=import
--import-source-dir=/var/lib/cassandra/download/sstables

echo " Lastly we can cleanup downloaded data as well as truncated"
echo " "
"$JAVA" -jar "./esop.jar"
restore
--storage-location=file:////var/lib/cassandra/backme
--data-directory=/var/lib/cassandra/data
--entities=test
--snapshot-tag=snappy123
--restoration-strategy-type=hardlinks
--restoration-phase-type=cleanup
--import-source-dir=/var/lib/cassandra/download/sstables`

[smiklosovic]

What is the content of "/var/lib/cassandra/data", list me directories of that directory please.

Internally, I am appending automatically "data" to "--data-directory" so if you specified "/var/lib/cassandra/data", I will append "data" to it so I am about to parse "/var/lib/cassandra/data/data" so I expect keyspace "test" to be there.

If you have "/var/lib/cassandra/data/test", then it can not find it I would say, so you might try "/var/lib/cassandra" instead as when I append "data" to it, it will be "/var/lib/cassandra/data/test" and that might be what you have.

[vbpatil449]

Thanks.. I have tried '--data-directory=/var/lib/cassandra' also - same error.
Here is my list

root@cassandra-0-seed:/var/lib/cassandra# ls -l
total 28
drwxr-xr-x 3 cassandra root 4096 Feb 10 22:19 backcommit
drwxr-xr-x 4 cassandra root 4096 Feb 10 22:12 backme
drwxr-xr-x 2 cassandra cassandra 4096 Feb 11 14:32 commitlog
drwxr-xr-x 7 cassandra cassandra 4096 Feb 10 22:21 data
drwxr-xr-x 2 cassandra cassandra 4096 Feb 10 22:21 hints
drwxr-xr-x 2 cassandra cassandra 4096 Feb 11 14:21 saved_caches
drwxr-xr-x 2 cassandra root 4096 Feb 10 22:48 scripts

root@cassandra-0-seed:/var/lib/cassandra# ls backme/
data manifests
root@cassandra-0-seed:/var/lib/cassandra# ls backcommit/
commitlog

root@cassandra-0-seed:/var/lib/cassandra# ls data
system system_auth system_distributed system_schema system_traces

[vbpatil449]

I think 'test' keyspace directory is missing but cqlsh shows it is there... not sure why

root@cassandra-0-seed:/var/lib/cassandra/data# cqlsh -e "desc keyspaces"
system_schema system_auth system system_distributed test system_traces

root@cassandra-0-seed:/var/lib/cassandra/data# ls
system system_auth system_distributed system_schema system_traces

[smiklosovic]

Yes so as I said, you have "--data-directory=/var/lib/cassandra/data", but I add one more "data" at the end so it is

--data-directory=/var/lib/cassandra/data/data

but you do not have it.

So setup only this

--data-directory=/var/lib/cassandra

(without "data" at the end).

The fact that you have test keyspace but it is not in data dir is very wrong. I would just start from scratch.