Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unauthorized user can drop system_auth from node without authorization #2346

Closed
abvgedeika opened this issue May 3, 2017 · 12 comments
Closed
Milestone

Comments

@abvgedeika
Copy link

abvgedeika commented May 3, 2017

Installation details
Scylla version (or git commit hash): 666.development-0.20170427.14b9aa2
OS (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu16.04

steps

  1. 2 nodes joined in the cluster. first node(127.0.0.1) is with authorization parameters in config:
    authenticator: org.apache.cassandra.auth.PasswordAuthenticator
    authorizer: org.apache.cassandra.auth.CassandraAuthorizer

the second one(127.0.0.2) without
2) cqlsh connection looks as expected:

$ cqlsh 127.0.0.1
Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Remote end requires authentication.',)})
$ cqlsh 127.0.0.1 -u cassandra -p cassandra
Connected to test at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.2.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh> 
[14]+  Stopped 

$ cqlsh 127.0.0.2
Connected to test at 127.0.0.2:9042.
[cqlsh 5.0.1 | Cassandra 2.2.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.

  1. with cqlsh connection to host without authorizer we are unauthorized to see users but able to drop system_auth
$ cqlsh 127.0.0.2
Connected to test at 127.0.0.2:9042.
[cqlsh 5.0.1 | Cassandra 2.2.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> LIST USERS ;
Unauthorized: Error from server: code=2100 [Unauthorized] message="You have to be logged in and not anonymous to perform this request"
cqlsh> DESCRIBE ;
Improper DESCRIBE command.
cqlsh> DESCRIBE KEYSPACEs;

system_traces  system_auth  system

cqlsh> drop keyspace system_auth;
OperationTimedOut: errors={'127.0.0.2': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=127.0.0.2
cqlsh> DESCRIBE KEYSPACEs;

system_traces  system

cqlsh> 

server's log:

INFO  2017-05-03 12:00:53,212 [shard 0] schema_tables - Dropping keyspace system_auth
INFO  2017-05-03 12:00:53,212 [shard 0] schema_tables - Dropping system_auth.credentials id=1064e0a0-1f2c-3a17-b803-cb8e23d8458f version=b464fb36-9757-3257-bd40-b8e7de736275
INFO  2017-05-03 12:00:53,212 [shard 0] schema_tables - Dropping system_auth.permissions id=2d324e48-3275-3517-8dd5-9a2c5b0856c5 version=6dfcbd32-c845-3656-9637-5d21b7d9db64
INFO  2017-05-03 12:00:53,212 [shard 0] schema_tables - Dropping system_auth.users id=473588ad-9c79-38be-8b59-e06c10456ba0 version=1bc55637-4b77-3a3f-9e43-44ba70e03331
INFO  2017-05-03 12:00:57,074 [shard 0] compaction - Compacting [/home/andrei/.dtest/dtest-EJiPxf/test/node1/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-8-Data.db:level=0, /home/andrei/.dtest/dtest-EJiPxf/test/node1/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-7-Data.db:level=0, /home/andrei/.dtest/dtest-EJiPxf/test/node1/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-6-Data.db:level=0, /home/andrei/.dtest/dtest-EJiPxf/test/node1/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-5-Data.db:level=0, ]
WARN  2017-05-03 12:00:57,074 [shard 0] migration_manager - Drop column family notification failed system_auth.credentials: exceptions::invalid_request_exception (unconfigured table permissions)
WARN  2017-05-03 12:00:57,074 [shard 0] migration_manager - Drop column family notification failed system_auth.permissions: exceptions::invalid_request_exception (unconfigured table permissions)
WARN  2017-05-03 12:00:57,075 [shard 0] migration_manager - Drop column family notification failed system_auth.users: exceptions::invalid_request_exception (unconfigured table permissions)
WARN  2017-05-03 12:00:57,075 [shard 0] migration_manager - Drop keyspace notification failed system_auth: exceptions::keyspace_not_defined_exception (Keyspace system_auth does not exist)
INFO  2017-05-03 12:00:57,190 [shard 0] database - Schema version changed to e3104b46-1e20-34c7-987e-2f542200930b
INFO  2017-05-03 12:00:57,576 [shard 0] compaction - Compacted 4 sstables to [/home/andrei/.dtest/dtest-EJiPxf/test/node1/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-9-Data.db:level=0, ]. 12998 bytes to 12869 (~99% of original) in 501ms = 0.02MB/s. ~1024 total partitions merged to 1.

In short, an unauthorized user from _authorizer node can break authorization for entire cluster

@slivne
Copy link
Contributor

slivne commented May 3, 2017 via email

@slivne slivne added this to the x-ray milestone May 4, 2017
@slivne
Copy link
Contributor

slivne commented May 4, 2017

@abvgedeika can you please provide the additional information

@abvgedeika
Copy link
Author

abvgedeika commented May 4, 2017

hi @slivne
in C* we can't drop system_auth at all, please see #2338

here are steps with the same scenarios in C*( 127.0.0.1 - with authorizer, 127.0.0.2 - without)

$ ./cassandra/bin/cqlsh 127.0.0.2
Connected to test at 127.0.0.2:9042.
[cqlsh 5.0.1 | Cassandra 3.0.9-SNAPSHOT | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cqlsh> LIST USERS ;
Unauthorized: Error from server: code=2100 [Unauthorized] message="You have to be logged in and not anonymous to perform this request"
cqlsh> DESCRIBE KEYSPACEs;

system_traces  system_schema  system_auth  system  system_distributed

cqlsh> drop keyspace system_auth;
Unauthorized: Error from server: code=2100 [Unauthorized] message="Cannot DROP <keyspace system_auth>"
cqlsh> DESCRIBE KEYSPACES;

system_traces  system_schema  system_auth  system  system_distributed

cqlsh> 
cqlsh> 
[5]+  Stopped                 ./cassandra/bin/cqlsh 127.0.0.2
$ ^C
$ 
$ ./cassandra/bin/cqlsh 127.0.0.1
Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Remote end requires authentication.',)})
$ ./cassandra/bin/cqlsh 127.0.0.1 -u cassandra -p cassandra
Connected to test at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.0.9-SNAPSHOT | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh>  LIST USERS ;

 name      | super
-----------+-------
 cassandra |  True

(1 rows)
cassandra@cqlsh> DESCRIBE KEYSPACES;

system_traces  system_schema  system_auth  system  system_distributed

cassandra@cqlsh> drop keyspace system_auth;
Unauthorized: Error from server: code=2100 [Unauthorized] message="Cannot DROP <keyspace system_auth>"

@vladzcloudius
Copy link
Contributor

@slivne @abvgedeika There is more - in C* one can't ALTER any system_auth or system_traces table. The means that if the corresponding table has changed (e.g. was added a new column) in the new release the upgrade is simply impossible and the whole cluster has to be rebuilt from scratch - which is crazy IMO.

I suggest we don't follow the C* pattern exactly but rather allow the ALTER command on tables. This means that we should only forbid the DROP command on both system_auth and system_traces keyspaces and all their tables.

@vladzcloudius
Copy link
Contributor

vladzcloudius commented May 9, 2017

@slivne @avi @abvgedeika In general this issue describes quite a serious security hole - this means that a malicious user may add a Node to the cluster (without authentication enabled) and bypass the authentication in the cluster. Is that right? If yes, then all authentication in scylla is a fake! Something is really fishy here...

@vladzcloudius
Copy link
Contributor

vladzcloudius commented May 9, 2017

I thought about it a little more - maybe it's not a fake but in order to ensure any protection the cluster has to be strongly protected against such unauthorized Nodes, e.g. by keeping all Nodes in a secure VPN.

@slivne, @tzach We have to make sure that this use case is very well described somewhere in our authentication documentation.

@tzach
Copy link
Contributor

tzach commented May 10, 2017

I suggest we don't follow the C* pattern exactly but rather allow the ALTER command on tables. This means that we should only forbid the DROP command on both system_auth and system_traces keyspaces and all their tables.

Why should we allow ALTER?
If the code assumes a system table schema, an ALTER will kill the code.

@abvgedeika
Copy link
Author

A little more information.

we are not able to CREATE/DROP/ALTER table from non-authorized node. we only can 'drop keyspace system_auth;'

$ cqlsh 127.0.0.2
Connected to test at 127.0.0.2:9042.
[cqlsh 5.0.1 | Cassandra 2.2.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> LIST USERS ;
Unauthorized: Error from server: code=2100 [Unauthorized] message="You have to be logged in and not anonymous to perform this request"
cqlsh> CREATE USER Test WITH PASSWORD '12345';
InvalidRequest: Error from server: code=2200 [Invalid query] message="org.apache.cassandra.auth.AllowAllAuthenticator doesn't support PASSWORD option"
cqlsh> ALTER USER Test WITH PASSWORD '54321';
Unauthorized: Error from server: code=2100 [Unauthorized] message="You aren't allowed to alter this user"
cqlsh> DROP USER Test;
Unauthorized: Error from server: code=2100 [Unauthorized] message="You have to be logged in and not anonymous to perform this request"

After adding non-authorized node and waiting a little, we can't connect to authorized node at all:

$ cqlsh 127.0.0.1 -u cassandra -p cassandra
Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Failed to authenticate to 127.0.0.1: Error from server: code=0100 [Bad credentials] message="authentication failed"',)})
$ cqlsh 127.0.0.1
Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Remote end requires authentication.',)})

@vladzcloudius
Copy link
Contributor

vladzcloudius commented May 10, 2017 via email

@vladzcloudius
Copy link
Contributor

vladzcloudius commented May 10, 2017 via email

@abvgedeika
Copy link
Author

@vladzcloudius
yes, ALTER TABLE allows to change system_auth.users

$ cqlsh 127.0.0.1
Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Remote end requires authentication.',)})
$ cqlsh 127.0.0.2
Connected to test at 127.0.0.2:9042.
[cqlsh 5.0.1 | Cassandra 2.2.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh>  alter table system_auth.users drop super ;
cqlsh> alter table system_auth.users add super boolean;
INFO  2017-05-10 21:20:21,210 [shard 0] schema_tables - Altering system_auth.users id=473588ad-9c79-38be-8b59-e06c10456ba0 version=df3c2cfc-7000-309b-8a85-137d593b08ff
INFO  2017-05-10 21:20:21,211 [shard 0] database - Setting compaction strategy of system_auth.users to SizeTieredCompactionStrategy
INFO  2017-05-10 21:20:21,245 [shard 0] query_processor - Column definitions for system_auth.users changed, invalidating related prepared statements
INFO  2017-05-10 21:20:21,248 [shard 0] database - Schema version changed to d10c292e-068a-3921-8119-5e9337eb0a53
INFO  2017-05-10 21:20:31,038 [shard 0] compaction - Compacting [/home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-15-Data.db:level=0, /home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-16-Data.db:level=0, /home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-14-Data.db:level=0, /home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-13-Data.db:level=0, ]
INFO  2017-05-10 21:20:32,339 [shard 0] compaction - Compacting [/home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-16-Data.db:level=0, /home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-15-Data.db:level=0, /home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-14-Data.db:level=0, /home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-13-Data.db:level=0, ]
INFO  2017-05-10 21:20:32,903 [shard 0] compaction - Compacted 4 sstables to [/home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-17-Data.db:level=0, ]. 45632 bytes to 44922 (~98% of original) in 1865ms = 0.02MB/s. ~1024 total partitions merged to 3.
INFO  2017-05-10 21:20:33,571 [shard 0] compaction - Compacted 4 sstables to [/home/andrei/.dtest/dtest-bteDt5/test/node1/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-17-Data.db:level=0, ]. 51333 bytes to 46779 (~91% of original) in 1231ms = 0.04MB/s. ~1024 total partitions merged to 3.
INFO  2017-05-10 21:20:33,572 [shard 0] schema_tables - Altering system_auth.users id=473588ad-9c79-38be-8b59-e06c10456ba0 version=0a7f2ca9-9a35-33dd-ae3c-95bc51a669f1
INFO  2017-05-10 21:20:33,573 [shard 0] database - Setting compaction strategy of system_auth.users to SizeTieredCompactionStrategy
INFO  2017-05-10 21:20:33,573 [shard 0] query_processor - Column definitions for system_auth.users changed, invalidating related prepared statements
INFO  2017-05-10 21:20:33,575 [shard 0] database - Schema version changed to fcb59483-d9fc-3b2d-9131-76cc7ef14609

@vladzcloudius
Copy link
Contributor

vladzcloudius commented May 10, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants