Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DTEST snapshot_test.py:TestSnapshot.test_basic_snapshot_and_restore_with_refresh not able to restore from a backup #488

Closed
slivne opened this issue Oct 22, 2015 · 5 comments
Closed

DTEST snapshot_test.py:TestSnapshot.test_basic_snapshot_and_restore_with_refresh not able to restore from a backup #488

slivne opened this issue Oct 22, 2015 · 5 comments
Assignees
@amnonh
Labels
type/bug

Comments

@slivne
Copy link
Contributor

slivne commented Oct 22, 2015 

shlomi@localhost~/urchin-dtest (snapshot_test_work_without_sstableloader)$ pkill -9 java ; KEEP_TEST_DIR=true DEBUG=true PRINT_DEBUG=true nosetests -v -s snapshot_test.py:TestSnapshot.test_basic_snapshot_and_restore_with_refresh
test_basic_snapshot_and_restore_with_refresh (snapshot_test.TestSnapshot) ... cluster ccm directory: /home/shlomi/.dtest/dtest-cI6JJ5
Making snapshot....
Running snapshot cmd: snapshot ks -cf cf -t basic
snapshot_dir is : /home/shlomi/.dtest/dtest-cI6JJ5/test/node1/data/ks/cf-1da4651078cd11e5b001000000000000/snapshots/basic
snapshot copy is : /tmp/tmpWuyr5W
Restoring snapshot....
Copying from /tmp/tmpWuyr5W/ks/cf to /home/shlomi/.dtest/dtest-cI6JJ5/test/node1/data/ks/cf-21a79dd078cd11e5b001000000000000/
ERROR

======================================================================
ERROR: test_basic_snapshot_and_restore_with_refresh (snapshot_test.TestSnapshot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shlomi/urchin-dtest/snapshot_test.py", line 84, in test_basic_snapshot_and_restore_with_refresh
    self.basic_snapshot_and_restore(use_sstableloader=False)
  File "/home/shlomi/urchin-dtest/snapshot_test.py", line 115, in basic_snapshot_and_restore
    self.restore_snapshot_with_refresh(snapshot_dir, node1, 'ks', 'cf')
  File "/home/shlomi/urchin-dtest/snapshot_test.py", line 72, in restore_snapshot_with_refresh
    node.nodetool("refresh %s %s" % (ks,cf))
  File "/home/shlomi/maintainer/urchin-ccm/ccmlib/node.py", line 595, in nodetool
    raise NodetoolError(" ".join(args), exit_status, stdout, stderr)
NodetoolError: Nodetool command '/home/shlomi/scylla/resources/cassandra/bin/nodetool -h localhost -p 7100 refresh ks cf' failed; exit status: 2; stderr: error: com.sun.jersey.api.client.ClientHandlerException (no security manager: RMI class loader disabled)
-- StackTrace --
java.lang.ClassNotFoundException: com.sun.jersey.api.client.ClientHandlerException (no security manager: RMI class loader disabled)
    at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396)
    at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:186)
    at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
    at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
    at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:214)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:245)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
    at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
    at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
    at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1022)
    at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
    at com.sun.proxy.$Proxy7.loadNewSSTables(Unknown Source)
    at org.apache.cassandra.tools.NodeProbe.loadNewSSTables(NodeProbe.java:998)
    at org.apache.cassandra.tools.NodeTool$Refresh.execute(NodeTool.java:1793)
    at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:288)
    at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)


-------------------- >> begin captured logging << --------------------
dtest: DEBUG: cluster ccm directory: /home/shlomi/.dtest/dtest-cI6JJ5
dtest: DEBUG: Making snapshot....
dtest: DEBUG: Running snapshot cmd: snapshot ks -cf cf -t basic
dtest: DEBUG: snapshot_dir is : /home/shlomi/.dtest/dtest-cI6JJ5/test/node1/data/ks/cf-1da4651078cd11e5b001000000000000/snapshots/basic
dtest: DEBUG: snapshot copy is : /tmp/tmpWuyr5W
dtest: DEBUG: Restoring snapshot....
dtest: DEBUG: Copying from /tmp/tmpWuyr5W/ks/cf to /home/shlomi/.dtest/dtest-cI6JJ5/test/node1/data/ks/cf-21a79dd078cd11e5b001000000000000/
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 12.276s

FAILED (errors=1)

it seems that nodetool refresh is not wirred up
@glommer
Copy link
Contributor

glommer commented Oct 22, 2015

I have no idea whether or not this is wired. @amnonh should check that and come back with info.

I only went as far as implementing (and testing) load_new_sstables in the json api

@slivne
Copy link
Contributor Author

slivne commented Nov 8, 2015

Translated the script - to debug the issue - the problem is the command nodetool refressh it works via the rest api

rm -Rf /tmp/basic/*
mkdir /tmp/basic

ccm remove
ccm create scylla-1 --urchin --vnodes -n 1 --install-dir=/home/shlomi/urchin-ccm/../scylla
ccm start

ccm node1 stress write n=10000 
ccm node1 flush
#./nodetool -h localhost -p 7100 snapshot keyspace1 -cf standard1 -t basic
ccm node1 nodetool snapshot "keyspace1 -cf standard1 -t basic"

# copy snapshot
cp  -Rf ~/.ccm/scylla-1/node1/data/keyspace1/*/snapshots/basic/* /tmp/basic/

# remove
ccm node1 cqlsh -e "drop keyspace keyspace1"
mv ~/.ccm/scylla-1/node1/data/keyspace1/standard1-* ~/.ccm/scylla-1/node1/data/keyspace1/standard1old

# recreate
ccm node1 cqlsh -e "CREATE KEYSPACE keyspace1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}  AND durable_writes = true;"
ccm node1 cqlsh -e 'CREATE TABLE keyspace1.standard1 ( key blob PRIMARY KEY, "C0" blob, "C1" blob, "C2" blob, "C3" blob, "C4" blob ) WITH COMPACT STORAGE;'

# try to restore
cp /tmp/basic/* ~/.ccm/scylla-1/node1/data/keyspace1/standard1-*
# bypass "{"message": "invalid version for file manifest.json. Name doesn't match any known version.", "code": 500}"
rm ~/.ccm/scylla-1/node1/data/keyspace1/standard1-*/manifest.json

now executing

ccm node1 nodetool refresh keyspace1 standard1

fails with

shlomi@localhost~/urchin-ccm (master)$ ccm node1 nodetool refresh keyspace1 standard1 
Traceback (most recent call last):
  File "/usr/bin/ccm", line 5, in <module>
    pkg_resources.run_script('ccm==2.0.3.1', 'ccm')
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 483, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1348, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/lib/python2.7/site-packages/ccm-2.0.3.1-py2.7.egg/EGG-INFO/scripts/ccm", line 72, in <module>

  File "/home/shlomi/urchin-ccm/ccmlib/cmds/node_cmds.py", line 237, in run
    stdout, stderr = self.node.nodetool(" ".join(self.args[1:]))
  File "/home/shlomi/urchin-ccm/ccmlib/node.py", line 598, in nodetool
    raise NodetoolError(" ".join(args), exit_status, stdout, stderr)
ccmlib.node.NodetoolError: Nodetool command '/home/shlomi/scylla/resources/cassandra/bin/nodetool -h localhost -p 7100 refresh keyspace1 standard1' failed; exit status: 2; stderr: error: com.sun.jersey.api.client.UniformInterfaceException (no security manager: RMI class loader disabled)
-- StackTrace --
java.lang.ClassNotFoundException: com.sun.jersey.api.client.UniformInterfaceException (no security manager: RMI class loader disabled)
    at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396)
    at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:186)
    at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
    at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
    at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:214)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:245)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
    at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
    at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
    at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1022)
    at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
    at com.sun.proxy.$Proxy7.loadNewSSTables(Unknown Source)
    at org.apache.cassandra.tools.NodeProbe.loadNewSSTables(NodeProbe.java:998)
    at org.apache.cassandra.tools.NodeTool$Refresh.execute(NodeTool.java:1793)
    at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:288)
    at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)

doing the same via rest api works

curl -X POST --header "Content-Type: application/json" --header "Accept: application/json" "http://127.0.0.1:10000/storage_service/sstables/keyspace1?cf=standard1" 

ccm node1 cqlsh -e "select count(*) from keyspace1.standard1"

works

@amnonh
Copy link
Contributor

amnonh commented Nov 8, 2015

I tried to reproduce the problem, does it still exists with the latest version?

@slivne
Copy link
Contributor Author

slivne commented Nov 8, 2015

ok pulled scylla-jmx and ran again it works

thanks

@slivne slivne closed this as completed Nov 8, 2015
@slivne
Copy link
Contributor Author

slivne commented Nov 8, 2015

we still need #529 - handling of unknown file names to be fixed (currently manifest.json may fail the test)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@amnonh amnonh

Labels
type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@glommer @amnonh @slivne