Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible network corruption issue #25128

Closed
jprellberg opened this issue Jul 27, 2016 · 4 comments
Closed

Possible network corruption issue #25128

jprellberg opened this issue Jul 27, 2016 · 4 comments
Labels
area/networking kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. platform/desktop version/1.12

Comments

@jprellberg
Copy link

I run a Cassandra database server inside a docker container. When accessing the database from Java I ran into issues which at first I thought was a bug in the library (com.datastax.cassandra:cassandra-driver-core:3.0.3) I used. After filing a bug report the issue was not reproducible for the maintainers. I myself conducted more tests and the issue seems to be specific to running Cassandra inside a Docker container on Windows.

Below you find a piece of code that works correctly when run against a local Cassandra installation on Windows 10 64-bit. When run against a Cassandra docker image (I tested with cassandra:3.0.8 and cassandra:3.7) the code throws an exception. The exception seems to be caused by network package corruption as indicated by a maintainer of the cassandra-driver-core library. The code itself just performs inserts into a table using batches.

Output of docker version:

C:\WINDOWS\system32>docker version
Client:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:28:51 2016
 OS/Arch:      windows/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc4
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   e4a0dbc
 Built:        Wed Jul 13 03:28:51 2016
 OS/Arch:      linux/amd64
 Experimental: true

Output of docker info:

C:\WINDOWS\system32>docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 81
Server Version: 1.12.0-rc4
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 43
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.15-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.951 GiB
Name: moby
ID: HUXC:VKG7:LQIQ:O2XK:R5Z4:47D6:ZV4C:Z3UT:6WA3:BE7H:ER5X:4IX2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Docker is running on Windows 10 64-bit.

Steps to reproduce the issue:

  1. Run a Cassandra database: docker run -d -p 9042:9042/tcp --name cassandra cassandra:3.7
  2. Run the following Java code against that database, compiled with the maven dependency com.datastax.cassandra:cassandra-driver-core:3.0.3:
package test;

import com.datastax.driver.core.BatchStatement;
import com.datastax.driver.core.BoundStatement;
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.PreparedStatement;
import com.datastax.driver.core.Session;

public class BatchTest {
    public static void main(String[] args) {
        Cluster cluster = null;
        Session session = null;
        try {
            System.out.println("Connecting...");
            cluster = Cluster.builder().addContactPoint("localhost").withPort(9042).build();
            session = cluster.connect();
            System.out.println("Connected.");
            session.execute(
                    "create keyspace if not exists test with replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };");
            System.out.println("Created keyspace.");
            session.execute("create table if not exists test.test (id int primary key);");
            System.out.println("Created table.");
            PreparedStatement prepStmt = session.prepare("insert into test.test (id) values (?);");
            BatchStatement batchStmt = new BatchStatement();
            for (int i = 0; i < 10000; i++) {
                BoundStatement boundStmt = new BoundStatement(prepStmt);
                boundStmt.setInt(0, i);
                batchStmt.add(boundStmt);
                if (batchStmt.size() >= 1000) {
                    session.execute(batchStmt);
                    batchStmt.clear();
                }
            }
        } finally {
            session.close();
            cluster.close();
        }
    }
}

Describe the results you received:
The program aborts with an exception.

Client side output:

Exception in thread "main" com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on localhost/127.0.0.1:9042: java.lang.IndexOutOfBoundsException: index: 4096, length: 1664221440 (expected: range(0, 45014))))
    at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)
    at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
    at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
    at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:245)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:64)
    at dbp.BatchTest4.main(BatchTest4.java:32)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on localhost/127.0.0.1:9042: java.lang.IndexOutOfBoundsException: index: 4096, length: 1664221440 (expected: range(0, 45014))))
    at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:207)
    at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:43)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.sendRequest(RequestHandler.java:273)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.run(RequestHandler.java:396)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

Server side output (docker logs cassandra)

INFO  14:42:54 Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  14:42:54 Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=172.17.0.2; broadcast_rpc_address=172.17.0.2; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@7219ec67; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=172.17.0.2; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.17.0.2}; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@45018215; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  14:42:54 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  14:42:54 Global memtable on-heap threshold is enabled at 244MB
INFO  14:42:54 Global memtable off-heap threshold is enabled at 244MB
WARN  14:42:54 Only 53.449GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  14:42:54 Hostname: 700330dec963
INFO  14:42:54 JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_91
INFO  14:42:54 Heap size: 978.000MiB/978.000MiB
INFO  14:42:54 Code Cache Non-heap memory: init = 2555904(2496K) used = 3559488(3476K) committed = 3604480(3520K) max = 251658240(245760K)
INFO  14:42:54 Metaspace Non-heap memory: init = 0(0K) used = 14979064(14627K) committed = 15597568(15232K) max = -1(-1K)
INFO  14:42:54 Compressed Class Space Non-heap memory: init = 0(0K) used = 1841192(1798K) committed = 1966080(1920K) max = 1073741824(1048576K)
INFO  14:42:54 Par Eden Space Heap memory: init = 167772160(163840K) used = 67192416(65617K) committed = 167772160(163840K) max = 167772160(163840K)
INFO  14:42:54 Par Survivor Space Heap memory: init = 20971520(20480K) used = 0(0K) committed = 20971520(20480K) max = 20971520(20480K)
INFO  14:42:54 CMS Old Gen Heap memory: init = 836763648(817152K) used = 0(0K) committed = 836763648(817152K) max = 836763648(817152K)
INFO  14:42:54 Classpath: /etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.36.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.3.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.3.jar:/usr/share/cassandra/lib/primitive-1.0.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.7.jar:/usr/share/cassandra/apache-cassandra-thrift-3.7.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar
INFO  14:42:54 JVM Arguments: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms998M, -Xmx998M, -Xmn200M, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes]
WARN  14:42:54 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  14:42:54 jemalloc shared library could not be preloaded to speed up memory allocations
WARN  14:42:54 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
WARN  14:42:54 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO  14:42:54 Initializing SIGAR library
WARN  14:42:54 Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true 
WARN  14:42:54 Directory /var/lib/cassandra/data doesn't exist
WARN  14:42:54 Directory /var/lib/cassandra/commitlog doesn't exist
WARN  14:42:54 Directory /var/lib/cassandra/saved_caches doesn't exist
WARN  14:42:54 Directory /var/lib/cassandra/hints doesn't exist
INFO  14:42:54 Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO  14:42:54 Initializing system.IndexInfo
INFO  14:42:55 Initializing system.batches
INFO  14:42:55 Initializing system.paxos
INFO  14:42:55 Initializing system.local
INFO  14:42:55 Initializing system.peers
INFO  14:42:55 Initializing system.peer_events
INFO  14:42:55 Initializing system.range_xfers
INFO  14:42:55 Initializing system.compaction_history
INFO  14:42:55 Initializing system.sstable_activity
INFO  14:42:55 Initializing system.size_estimates
INFO  14:42:55 Initializing system.available_ranges
INFO  14:42:55 Initializing system.views_builds_in_progress
INFO  14:42:55 Initializing system.built_views
INFO  14:42:55 Initializing system.hints
INFO  14:42:55 Initializing system.batchlog
INFO  14:42:55 Initializing system.schema_keyspaces
INFO  14:42:55 Initializing system.schema_columnfamilies
INFO  14:42:55 Initializing system.schema_columns
INFO  14:42:55 Initializing system.schema_triggers
INFO  14:42:55 Initializing system.schema_usertypes
INFO  14:42:55 Initializing system.schema_functions
INFO  14:42:55 Initializing system.schema_aggregates
INFO  14:42:55 Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  14:42:55 Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
INFO  14:42:55 Initializing key cache with capacity of 48 MBs.
INFO  14:42:55 Initializing row cache with capacity of 0 MBs
INFO  14:42:55 Initializing counter cache with capacity of 24 MBs
INFO  14:42:55 Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  14:42:56 Global buffer pool is enabled, when pool is exahusted (max is 244.000MiB) it will allocate on heap
INFO  14:42:56 Populating token metadata from system tables
INFO  14:42:56 Token metadata: 
INFO  14:42:56 Initializing system_schema.keyspaces
INFO  14:42:56 Initializing system_schema.tables
INFO  14:42:56 Initializing system_schema.columns
INFO  14:42:56 Initializing system_schema.triggers
INFO  14:42:56 Initializing system_schema.dropped_columns
INFO  14:42:56 Initializing system_schema.views
INFO  14:42:56 Initializing system_schema.types
INFO  14:42:56 Initializing system_schema.functions
INFO  14:42:56 Initializing system_schema.aggregates
INFO  14:42:56 Initializing system_schema.indexes
INFO  14:42:56 Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO  14:42:56 Completed loading (1 ms; 8 keys) KeyCache cache
INFO  14:42:56 No commitlog files found; skipping replay
INFO  14:42:56 Populating token metadata from system tables
INFO  14:42:56 Token metadata: 
INFO  14:42:56 Cassandra version: 3.7
INFO  14:42:56 Thrift API version: 20.1.0
INFO  14:42:56 CQL supported versions: 3.4.2 (default: 3.4.2)
INFO  14:42:56 Initializing index summary manager with a memory pool size of 48 MB and a resize interval of 60 minutes
INFO  14:42:56 Starting Messaging Service on /172.17.0.2:7000 (eth0)
WARN  14:42:56 No host ID found, created 003dde17-2140-45b1-8a4b-459a927689f4 (Note: This should happen exactly once per node).
INFO  14:42:56 Loading persisted ring state
INFO  14:42:56 Starting up server gossip
INFO  14:42:56 This node will not auto bootstrap because it is configured to be a seed node.
INFO  14:42:56 Generated random tokens. tokens are [1959721115509618690, -3062717389819775012, 4020329474181260042, -4570500773765344445, -106635158741936143, -7671377862514718982, -477061516195326636, -473225211287528646, 8523120202296288058, -6791280352271147642, 7994824193790670408, -5274119443644970980, -6215984912386784521, 9215310061616922009, 8801496264496062781, 3469888622763020445, 4648143478988753772, -1065312133740406249, 4344864664376740954, -294289200704739191, -8053425721227156305, 8829017091309272505, 6941471510613749078, -651304077631559005, 4889647241326635413, -3513504887205485728, -8335050677102300794, -2955387586540465979, -619087972208444843, 1336117526098025551, 7297652479993200905, 2723507331928060479, 3481587273170363027, -8926181100525718037, -7882889672192319393, -5507598950493349187, 9057923750842449872, 4734629918407255224, 1772023432805736255, 4863138612956232516, 3778377193803363896, 7164743948815366222, 5770833240478553253, 5902548245479278724, 8873105249406849226, -4867501581847680145, 3121908756671043827, -1321248224151644898, -3361645978737549966, -5534225270397800211, -5215267257601802203, -9112797431125812875, 8504613689010422315, 9129963561010627035, 800981757154179434, -4556790685458571064, -1562995132161872946, 3525701534384596305, 2230110768670031148, -8002973415515524669, 6017422297697989509, 3209632173857669109, 1824396790389438109, -3875080435293396766, -5928475873344063879, 530606563068887344, -7398895154095513575, 4842270403951745305, 4050895848829847392, 6063776227047810141, -7425438502865199034, 2047933601846373773, 5916889336128220716, -2346769607657130121, -3792611437211169825, -3417676594876472315, -3032813333147396988, -5091974736796991033, -5416147436088557036, -5629939385831180738, 7898923647023211760, -3908080377754967301, 3472374777916789272, 6382131528993199498, 5524058044080090503, -2210686601431906780, 2615568849124341579, -1139939293449936982, 4698289948610886898, -7436976322746982748, 3236478456529564896, -6697059751920302789, 6239169516166578484, 1847360388638546816, -2862822815280207503, -6903014900937415066, 9194549744721938089, 8719321889299428173, -5074122783978962130, -8127148082854877672, -669509910608302423, 2258155161528594707, 203765572990066723, 6496022209593261811, 6897308267545034048, -881430214171480851, 7058669206672115630, -7672986412842061077, -5598659371620477866, 8556167791181941054, 4687473048667445804, -147140062960030508, -3703885294096580304, 4839853518078589221, 2778057443700693937, 622659750053324820, 547638761310404838, -4843203659810072613, -7444925793559794013, -5826636850828154249, 7970446258785424603, 8615897316001847305, 7726444599211858801, -2552354755885214707, 6879084131476222886, -1051063655481518564, -2865145582839244155, 3777660681156127901, 4079761732327879804, -2683799194309483937, 232994321720221699, 4325732031485212304, -8957950336357583571, -1533783481030385510, -4896304332268197588, -5316711777192655548, -7694412144812543532, -3580686959307591679, -2188873871005198029, 5857479448101528075, -5753592129392602689, -390712168948361893, -2464418866149771660, 5082155762470617982, -7900112971768242165, -1515480295494666838, -4131959906556962701, -1789918826091646364, -4899301299099726363, 2469918419937672296, -2213912309232586649, -354109317888251157, -1957879526447018965, -5232149409138869969, 2880143496946387973, 2310215396222036937, 5873048274950487026, -2924308512342686591, -4973208204769872895, -6007238799923766733, 3105663653415890215, 2630207512597765514, 4491276957834952281, 5008516717121497214, -5772463383062308244, -1963437300556239793, 5241809658707910371, 1883185303013342282, -5590222122073024516, -5676671935102976184, 3988094864327972329, -391992374243580339, -6614357944783940902, -2980628319855329240, 7675515114090316028, -4473419338643860820, -530021539606511691, 1901548990901709375, 5211148215700822644, -8776678413768529044, 157831677620098209, 2942457617893032253, 4563513588506819015, -1271935319180204973, 6079165145960403104, -4007984572163925344, -7984145335968156989, -477595907457822476, 1047976644299110027, 2421232453132107638, -2014702879251843807, -451368242491526318, -1509166177895813544, -2532896630812012050, 7813763393249738490, -1321031847525423130, -2451779004956223915, -2756625952630147669, -8320845817520520144, 7385892291577374431, 2375242119177853504, 4019030221654088121, 4127954726700168174, -7569096720289813686, -7746120668482190698, -6870649487214986800, -3556213075559913061, 1431583206977414493, -3329936504160638767, -6539760970151426431, 1229747816832367114, -1356412511009828142, 3643579048997195374, -6616303950446769012, 6129554523358140879, 1793975509745284204, -189763379974541081, 1456635024357773935, -2590214914587098393, 7966842018771369766, 1089463912369344864, 1441843931669824618, 660201836045886230, -5977849465969825476, -4597937687739562656, -7252767921231350223, -5211561029577203079, -5742644063878798128, -8486305042460577284, -8304985239223746144, 3480740222260128375, -760262386711643606, 550182343766695525, 8085912039978637354, -7023424148160177756, -4130812412055451870, 1954779263927587628, -5216784559678640440, -3123022154660424868, 5128364815440321157, -9090246661647128262, 270426734972891738, 5804450997545679573, 5557114684943433344, -8414270377779371229, 7350849434593785122, 8734689308677592159, -3791695534876493620, 186906916594203141, -5677163471778849299, 2470589199892662768, -8358358573443223985, -1434937890618697068, 5442187528740623713, 934687906353631422, 7038755307293012705]
INFO  14:42:56 Create new Keyspace: KeyspaceMetadata{name=system_traces, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}, tables=[org.apache.cassandra.config.CFMetaData@2b1092ce[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,flags=[COMPOUND],params=TableParams{comment=tracing sessions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(),partitionColumns=[[] | [client command coordinator duration request started_at parameters]],partitionKeyColumns=[ColumnDefinition{name=session_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, position=0}],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[ColumnDefinition{name=client, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, position=-1}, ColumnDefinition{name=command, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=session_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=coordinator, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, position=-1}, ColumnDefinition{name=request, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, position=-1}, ColumnDefinition{name=duration, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, position=-1}, ColumnDefinition{name=parameters, type=org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, position=-1}],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@160064ae[cfId=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksName=system_traces,cfName=events,flags=[COMPOUND],params=TableParams{comment=tracing events, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [activity source source_elapsed thread]],partitionKeyColumns=[ColumnDefinition{name=session_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, position=0}],clusteringColumns=[ColumnDefinition{name=event_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=CLUSTERING, position=0}],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[ColumnDefinition{name=activity, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=session_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=thread, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=event_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=CLUSTERING, position=0}, ColumnDefinition{name=source, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, position=-1}, ColumnDefinition{name=source_elapsed, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, position=-1}],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  14:42:56 Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO  14:42:56 Initializing system_traces.events
INFO  14:42:56 Initializing system_traces.sessions
INFO  14:42:56 Create new Keyspace: KeyspaceMetadata{name=system_distributed, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}, tables=[org.apache.cassandra.config.CFMetaData@6636c83b[cfId=759fffad-624b-3181-80ee-fa9a52d1f627,ksName=system_distributed,cfName=repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [coordinator exception_message exception_stacktrace finished_at parent_id range_begin range_end started_at status participants]],partitionKeyColumns=[ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=columnfamily_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=1}],clusteringColumns=[ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=CLUSTERING, position=0}],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[ColumnDefinition{name=status, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=CLUSTERING, position=0}, ColumnDefinition{name=coordinator, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, position=-1}, ColumnDefinition{name=finished_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, position=-1}, ColumnDefinition{name=participants, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.InetAddressType), kind=REGULAR, position=-1}, ColumnDefinition{name=exception_stacktrace, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=parent_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=REGULAR, position=-1}, ColumnDefinition{name=range_end, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=range_begin, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=exception_message, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, position=-1}, ColumnDefinition{name=columnfamily_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=1}],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@11b8b037[cfId=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksName=system_distributed,cfName=parent_repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(),partitionColumns=[[] | [exception_message exception_stacktrace finished_at keyspace_name started_at columnfamily_names options requested_ranges successful_ranges]],partitionKeyColumns=[ColumnDefinition{name=parent_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=PARTITION_KEY, position=0}],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.TimeUUIDType,columnMetadata=[ColumnDefinition{name=requested_ranges, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, position=-1}, ColumnDefinition{name=exception_message, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=successful_ranges, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, position=-1}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, position=-1}, ColumnDefinition{name=finished_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, position=-1}, ColumnDefinition{name=options, type=org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, position=-1}, ColumnDefinition{name=exception_stacktrace, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=parent_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=columnfamily_names, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, position=-1}],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@38b7e79b[cfId=5582b59f-8e4e-35e1-b913-3acada51eb04,ksName=system_distributed,cfName=view_build_status,flags=[COMPOUND],params=TableParams{comment=Materialized View build status, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[] | [status]],partitionKeyColumns=[ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=view_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=1}],clusteringColumns=[ColumnDefinition{name=host_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=CLUSTERING, position=0}],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[ColumnDefinition{name=status, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=view_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=1}, ColumnDefinition{name=host_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=CLUSTERING, position=0}],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  14:42:56 Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
INFO  14:42:56 Initializing system_distributed.parent_repair_history
INFO  14:42:56 Initializing system_distributed.repair_history
INFO  14:42:56 Initializing system_distributed.view_build_status
INFO  14:42:56 Node /172.17.0.2 state jump to NORMAL
INFO  14:42:56 Create new Keyspace: KeyspaceMetadata{name=system_auth, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[org.apache.cassandra.config.CFMetaData@11f631dd[cfId=5bc52802-de25-35ed-aeab-188eecebb090,ksName=system_auth,cfName=roles,flags=[COMPOUND],params=TableParams{comment=role definitions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(),partitionColumns=[[] | [can_login is_superuser salted_hash member_of]],partitionKeyColumns=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=salted_hash, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, position=-1}, ColumnDefinition{name=member_of, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, position=-1}, ColumnDefinition{name=can_login, type=org.apache.cassandra.db.marshal.BooleanType, kind=REGULAR, position=-1}, ColumnDefinition{name=is_superuser, type=org.apache.cassandra.db.marshal.BooleanType, kind=REGULAR, position=-1}],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@3ba3aea1[cfId=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksName=system_auth,cfName=role_members,flags=[COMPOUND],params=TableParams{comment=role memberships lookup table, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}],clusteringColumns=[ColumnDefinition{name=member, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING, position=0}],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=member, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING, position=0}],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@2d489515[cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksName=system_auth,cfName=role_permissions,flags=[COMPOUND],params=TableParams{comment=permissions granted to db roles, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | [permissions]],partitionKeyColumns=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}],clusteringColumns=[ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING, position=0}],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING, position=0}, ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=permissions, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, position=-1}],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@7f637d0b[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,flags=[COMPOUND],params=TableParams{comment=index of db roles with permissions granted on a resource, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}],clusteringColumns=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING, position=0}],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}, ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING, position=0}],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  14:42:56 Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
INFO  14:42:56 Initializing system_auth.resource_role_permissons_index
INFO  14:42:56 Initializing system_auth.role_members
INFO  14:42:56 Initializing system_auth.role_permissions
INFO  14:42:56 Initializing system_auth.roles
INFO  14:42:56 Waiting for gossip to settle before accepting client requests...
INFO  14:43:04 No gossip backlog; proceeding
INFO  14:43:04 Netty using native Epoll event loop
INFO  14:43:05 Using Netty Version: [netty-buffer=netty-buffer-4.0.36.Final.e8fa848, netty-codec=netty-codec-4.0.36.Final.e8fa848, netty-codec-haproxy=netty-codec-haproxy-4.0.36.Final.e8fa848, netty-codec-http=netty-codec-http-4.0.36.Final.e8fa848, netty-codec-socks=netty-codec-socks-4.0.36.Final.e8fa848, netty-common=netty-common-4.0.36.Final.e8fa848, netty-handler=netty-handler-4.0.36.Final.e8fa848, netty-tcnative=netty-tcnative-1.1.33.Fork15.906a8ca, netty-transport=netty-transport-4.0.36.Final.e8fa848, netty-transport-native-epoll=netty-transport-native-epoll-4.0.36.Final.e8fa848, netty-transport-rxtx=netty-transport-rxtx-4.0.36.Final.e8fa848, netty-transport-sctp=netty-transport-sctp-4.0.36.Final.e8fa848, netty-transport-udt=netty-transport-udt-4.0.36.Final.e8fa848]
INFO  14:43:05 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO  14:43:05 Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO  14:43:06 Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  14:43:07 Created default superuser role 'cassandra'
INFO  14:43:38 Create new Keyspace: KeyspaceMetadata{name=batchtest, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[], views=[], functions=[], types=[]}
INFO  14:43:38 Create new table: org.apache.cassandra.config.CFMetaData@451fd8e0[cfId=81559540-5408-11e6-8376-6d2c86545d91,ksName=batchtest,cfName=test,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@de21c84c, extensions={}},comparator=comparator(org.apache.cassandra.db.marshal.LongType, org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[ColumnDefinition{name=c0, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}],clusteringColumns=[ColumnDefinition{name=c1, type=org.apache.cassandra.db.marshal.LongType, kind=CLUSTERING, position=0}, ColumnDefinition{name=c2, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING, position=1}],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[ColumnDefinition{name=c2, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING, position=1}, ColumnDefinition{name=c1, type=org.apache.cassandra.db.marshal.LongType, kind=CLUSTERING, position=0}, ColumnDefinition{name=c0, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, position=0}],droppedColumns={},triggers=[],indexes=[]]
INFO  14:43:38 Initializing batchtest.test
ERROR 14:43:38 Unexpected exception during request; channel = [id: 0x5f98c3be, L:/172.17.0.2:9042 - R:/172.17.0.1:41208]
java.lang.IndexOutOfBoundsException: index: 4096, length: 1664221440 (expected: range(0, 45014))
    at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1142) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.buffer.SlicedByteBuf.slice(SlicedByteBuf.java:155) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.buffer.AbstractByteBuf.readSlice(AbstractByteBuf.java:686) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at org.apache.cassandra.transport.CBUtil.readBoundValue(CBUtil.java:405) ~[apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.transport.CBUtil.readValueList(CBUtil.java:462) ~[apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.transport.messages.BatchMessage$1.decode(BatchMessage.java:61) ~[apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.transport.messages.BatchMessage$1.decode(BatchMessage.java:45) ~[apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.transport.Message$ProtocolDecoder.decode(Message.java:280) ~[apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.transport.Message$ProtocolDecoder.decode(Message.java:261) ~[apache-cassandra-3.7.jar:3.7]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:277) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:372) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:245) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:962) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:879) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:360) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.0.36.Final.jar:4.0.36.Final]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]

Describe the results you expected:

The program should terminate normally.

Additional information you deem important (e.g. issue happens only occasionally):

When modifying the code to use smaller batch sizes, e.g.

if (batchStmt.size() >= 10) {
                    session.execute(batchStmt);
                    batchStmt.clear();
}

everything works.
@thaJeztah
Copy link
Member

ping @justincormack could this be a Docker for Windows issue?

@thaJeztah thaJeztah added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. area/networking labels Jul 27, 2016
@friism
Copy link
Contributor

friism commented Jul 27, 2016

@thaJeztah sounds very likely - there's a fix going out this week.

@justincormack
Copy link
Contributor

Yes it seems likely, there is a fix for a similar issue it is likely the same.

@friism
Copy link
Contributor

friism commented Aug 7, 2016

This should be fixed now: https://docs.docker.com/docker-for-windows/release-notes/#docker-for-windows-1-12-0-2016-07-28-1-12-0-rc4-beta21

Recommend closing unless it still reproduces

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. platform/desktop version/1.12
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@friism @justincormack @thaJeztah @GordonTheTurtle @jprellberg