Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch refuses to work when listen_address differs from broadcast_address #183

Closed
zllovesuki opened this issue Apr 8, 2018 · 21 comments

Comments

@zllovesuki
Copy link

I'm using Elassandra 5.5.0.14 (installed via .deb), and I setup up an 3-node cluster. However, when I try to run the example:

$ curl -XPUT 'http://whirlpool-elassandra-fmt01-01-15-1.f3w.net:9200/twitter/doc/1?pretty' -H 'Content-Type: application/json' -d '
> {
>     "user": "Poulpy",
>     "post_date": "2017/10/4 13:12:00",
>     "message": "Elassandra adds dynamic mapping to Cassandra"
> }'

The terminal is stuck, and the system.log emits:

2018-04-08 09:01:46,502 INFO  [elasticsearch[64.71.145.11][clusterService#updateTask][T#1]] MigrationManager.java:331 announceNewKeyspace Create new Keyspace: KeyspaceMetadata{name=twitter, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, fmt01=1}}, tables=[], views=[], functions=[], types=[]}
2018-04-08 09:01:50,264 WARN  [elasticsearch[64.71.145.11][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-08 09:01:50,273 ERROR [elasticsearch[64.71.145.11][clusterService#updateTask][T#1]] ClusterService.java:1542 publishAndApplyChanges Cassandra issue:
java.lang.NullPointerException: null
        at org.elassandra.discovery.CassandraDiscovery.isNormal(CassandraDiscovery.java:519)
        at org.elassandra.discovery.CassandraDiscovery.access$900(CassandraDiscovery.java:90)
        at org.elassandra.discovery.CassandraDiscovery$MetaDataVersionAckListener.lambda$new$0(CassandraDiscovery.java:479)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at org.elassandra.discovery.CassandraDiscovery$MetaDataVersionAckListener.<init>(CassandraDiscovery.java:476)
        at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:1487)
        at org.elasticsearch.cluster.service.BaseClusterService.runTasks(BaseClusterService.java:583)
        at org.elasticsearch.cluster.service.BaseClusterService$ClusterServiceTaskBatcher.run(BaseClusterService.java:263)
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
        at org.elasticsearch.cluster.service.BaseClusterService$ClusterServiceTaskBatcher$UpdateTask.run(BaseClusterService.java:266)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Eventually, curl returns an error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "resource.type" : "index_expression",
        "resource.id" : "twitter",
        "index_uuid" : "_na_",
        "index" : "twitter"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "resource.type" : "index_expression",
    "resource.id" : "twitter",
    "index_uuid" : "_na_",
    "index" : "twitter"
  },
  "status" : 404
}

/_cluster/state:

{
  "cluster_name" : "Whirlpool",
  "version" : 10,
  "state_uuid" : "UY-PxoWiTIGw6xrSep9jLw",
  "master_node" : "81923937-6b7b-4f4e-836c-728a69aa50f3",
  "blocks" : { },
  "nodes" : {
    "a9ade22a-4ffb-47bd-9087-c8e635af995c" : {
      "name" : "2001:470:1:298:c:a:a:0",
      "status" : "ALIVE",
      "ephemeral_id" : "a9ade22a-4ffb-47bd-9087-c8e635af995c",
      "transport_address" : "64.71.145.10:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "3be98785-40dc-44d9-b13f-06603e4f0f5d" : {
      "name" : "2001:470:1:298:c:a:a:1",
      "status" : "ALIVE",
      "ephemeral_id" : "3be98785-40dc-44d9-b13f-06603e4f0f5d",
      "transport_address" : "64.71.145.11:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "81923937-6b7b-4f4e-836c-728a69aa50f3" : {
      "name" : "64.71.145.12",
      "status" : "ALIVE",
      "ephemeral_id" : "81923937-6b7b-4f4e-836c-728a69aa50f3",
      "transport_address" : "64.71.145.12:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 0,
    "cluster_uuid" : "a9ade22a-4ffb-47bd-9087-c8e635af995c",
    "templates" : { },
    "indices" : { },
    "index-graveyard" : {
      "tombstones" : [ ]
    }
  },
  "routing_table" : {
    "indices" : { }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "3be98785-40dc-44d9-b13f-06603e4f0f5d" : [ ],
      "a9ade22a-4ffb-47bd-9087-c8e635af995c" : [ ],
      "81923937-6b7b-4f4e-836c-728a69aa50f3" : [ ]
    }
  }
}

nodetool describecluster:

# nodetool describecluster
Cluster Information:
        Name: Whirlpool
        Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
        DynamicEndPointSnitch: enabled
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
                3d4b194d-69e2-3d97-b0d0-ae36e5587409: [2001:470:1:298:c:a:a:0, 2001:470:1:298:c:a:a:1, 2001:470:1:298:c:a:a:2]

Please advise.

@vroyer
Copy link
Collaborator

vroyer commented Apr 8, 2018 via email

@zllovesuki
Copy link
Author

In the YAML, I have listen_address set to v4, broadcast set to v6 (listen on broadcast is true). And rpc_address is v4. However, I overloaded the elasticsearch conf transport.(bind_host, host, and publish_host) with v4 address.

Sounds messy, I know :P

@zllovesuki
Copy link
Author

zllovesuki commented Apr 8, 2018

Although, netstat does show that the nodes were connected to each other via :9300 on the v4 address.

@vroyer
Copy link
Collaborator

vroyer commented Apr 8, 2018 via email

@zllovesuki
Copy link
Author

nodetool gossipinfo yields:

/2001:470:1:298:c:a:a:0
  generation:1523209440
  heartbeat:43
  STATUS:16:NORMAL,-1252168997441022590
  LOAD:20:336031.0
  SCHEMA:12:de5dcc37-479a-3829-b955-e0067e77b78e
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:64.71.145.10
  RPC_ADDRESS:3:64.71.145.10
  NET_VERSION:1:11
  HOST_ID:2:a9ade22a-4ffb-47bd-9087-c8e635af995c
  RPC_READY:32:true
  X1:34:{}
  X2:36:a9ade22a-4ffb-47bd-9087-c8e635af995c/0
  TOKENS:15:<hidden>
/2001:470:1:298:c:a:a:1
  generation:1523209255
  heartbeat:238
  STATUS:16:NORMAL,-2713670237890558283
  LOAD:219:267937.0
  SCHEMA:84:de5dcc37-479a-3829-b955-e0067e77b78e
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:64.71.145.11
  RPC_ADDRESS:3:64.71.145.11
  NET_VERSION:1:11
  HOST_ID:2:3be98785-40dc-44d9-b13f-06603e4f0f5d
  RPC_READY:32:true
  X1:35:{}
  X2:37:a9ade22a-4ffb-47bd-9087-c8e635af995c/0
  TOKENS:15:<hidden>
/2001:470:1:298:c:a:a:2
  generation:1523208853
  heartbeat:660
  STATUS:16:NORMAL,-1547552548766245654
  LOAD:659:281869.0
  SCHEMA:503:de5dcc37-479a-3829-b955-e0067e77b78e
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:64.71.145.12
  RPC_ADDRESS:3:64.71.145.12
  NET_VERSION:1:11
  HOST_ID:2:81923937-6b7b-4f4e-836c-728a69aa50f3
  RPC_READY:32:true
  X1:34:{}
  X2:36:a9ade22a-4ffb-47bd-9087-c8e635af995c/0
  TOKENS:15:<hidden>

@vroyer
Copy link
Collaborator

vroyer commented Apr 11, 2018

Well, right now, i'don't have any solution to lookup gossip state from the ipv4 RPC address when nodes are using an ipv6 address like that (it would require a o(n) search in the gossip state map in CassandraDiscover:479). Could you fix your configuration to make it works ?
I keep this for further improvements.

@zllovesuki
Copy link
Author

^ The rationale behind this was to force IPv6 for inter-DC communication and limit IPv4 to just inter-node communication within the same datacenter. Is that something I can do with Elassandra?

@zllovesuki
Copy link
Author

@vinodc I've managed to get it working. However, running the example yields this:

2018-04-26 06:54:17,418 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] StorageService.java:3846 setLoggingLevel set log level to TRACE for classes under 'index.search.slowlog.query.twitter' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
2018-04-26 06:54:17,418 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] StorageService.java:3846 setLoggingLevel set log level to TRACE for classes under 'index.search.slowlog.fetch.twitter' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
2018-04-26 06:54:17,426 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] MigrationManager.java:331 announceNewKeyspace Create new Keyspace: KeyspaceMetadata{name=twitter, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, fmt01=1}}, tables=[], views=[], functions=[], types=[]}
2018-04-26 06:54:17,458 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:17,472 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:17,472 INFO  [metadataSchemaUpdater] MigrationManager.java:457 announceColumnFamilyUpdate Update table 'elastic_admin/metadata' From org.apache.cassandra.config.CFMetaData@62a0fa6d[cfId=48c58910-491b-11e8-b66a-6f7c187a5cec,ksName=elastic_admin,cfName=metadata,flags=[COMPOUND],params=TableParams{comment={"meta-data":{"version":6,"cluster_uuid":"3a2236ad-bbe5-4d7a-9956-c425e09e91a8","templates":{},"index-graveyard":{"tombstones":[{"index":{"index_name":"twitter","index_uuid":"TF9zOFiJSXWTjDK8Tf6tnQ"},"delete_date_in_millis":1524724789970},{"index":{"index_name":"twitter","index_uuid":"Ox4WhQRST1Owbn4OnWITLQ"},"delete_date_in_millis":1524724847859},{"index":{"index_name":"twitter","index_uuid":"KBgt4IAnQqKq9en9GiXG9w"},"delete_date_in_millis":1524724946291}]}}}, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={max_threshold=32, min_threshold=4}}, compression=org.apache.cassandra.schema.CompressionParams@a32a3c38, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [metadata owner version]],partitionKeyColumns=[cluster_name],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[owner, version, cluster_name, metadata],droppedColumns={},triggers=[],indexes=[]] To org.apache.cassandra.config.CFMetaData@694d9f5f[cfId=48c58910-491b-11e8-b66a-6f7c187a5cec,ksName=elastic_admin,cfName=metadata,flags=[COMPOUND],params=TableParams{comment={"meta-data":{"version":7,"cluster_uuid":"de73532a-94f6-45df-85fd-844270a3151a","templates":{},"indices":{"twitter":{"version":1,"state":"open","settings":{"index.creation_date":"1524725657417","index.provided_name":"twitter","index.uuid":"VVxdin1YQyS172wsv6MiNg","index.version.created":"5050099"},"mappings":[],"aliases":{}}},"index-graveyard":{"tombstones":[{"index":{"index_name":"twitter","index_uuid":"TF9zOFiJSXWTjDK8Tf6tnQ"},"delete_date_in_millis":1524724789970},{"index":{"index_name":"twitter","index_uuid":"Ox4WhQRST1Owbn4OnWITLQ"},"delete_date_in_millis":1524724847859},{"index":{"index_name":"twitter","index_uuid":"KBgt4IAnQqKq9en9GiXG9w"},"delete_date_in_millis":1524724946291}]}}}, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={max_threshold=32, min_threshold=4}}, compression=org.apache.cassandra.schema.CompressionParams@a32a3c38, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [metadata owner version]],partitionKeyColumns=[cluster_name],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[owner, version, cluster_name, metadata],droppedColumns={},triggers=[],indexes=[]]
2018-04-26 06:54:17,473 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] StorageService.java:3846 setLoggingLevel set log level to TRACE for classes under 'index.search.slowlog.query.twitter' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
2018-04-26 06:54:17,473 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] StorageService.java:3846 setLoggingLevel set log level to TRACE for classes under 'index.search.slowlog.fetch.twitter' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
2018-04-26 06:54:17,478 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:18,748 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] ClusterService.java:1705 publishAndApplyChanges failed to apply updated cluster state in 1.2s:
version [51], uuid [qnsWmg3GTsCxN14PKFPkvg], source [create-index [twitter], cause [auto(bulk api)]]
nodes: 
   {172.16.0.100}{de73532a-94f6-45df-85fd-844270a3151a}{de73532a-94f6-45df-85fd-844270a3151a}{172.16.0.100}{172.16.0.100:9300}{ALIVE}{rack=01-15, dc=fmt01}, local, master
   {64.62.210.135}{b27dbc1f-7e23-4d12-acaf-75e41ad523a1}{b27dbc1f-7e23-4d12-acaf-75e41ad523a1}{64.62.210.135}{64.62.210.135:9300}{ALIVE}{rack=01-15, dc=fmt01}
   {65.49.34.153}{3a2236ad-bbe5-4d7a-9956-c425e09e91a8}{3a2236ad-bbe5-4d7a-9956-c425e09e91a8}{65.49.34.153}{65.49.34.153:9300}{ALIVE}{rack=01-15, dc=fmt01}
routing_table (version 0):
routing_nodes:
-----node_id[de73532a-94f6-45df-85fd-844270a3151a][V]
-----node_id[b27dbc1f-7e23-4d12-acaf-75e41ad523a1][V]
-----node_id[3a2236ad-bbe5-4d7a-9956-c425e09e91a8][V]
---- unassigned

java.lang.AssertionError: null
	at org.elasticsearch.action.support.ActiveShardCount.enoughShardsActive(ActiveShardCount.java:179)
	at org.elasticsearch.action.support.ActiveShardsObserver.waitForActiveShards(ActiveShardsObserver.java:74)
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.lambda$createIndex$1(MetaDataCreateIndexService.java:208)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
	at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAllNodesAcked(AckedClusterStateUpdateTask.java:69)
	at org.elasticsearch.cluster.service.BaseClusterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(BaseClusterService.java:1007)
	at org.elasticsearch.cluster.service.BaseClusterService$AckCountDownListener.onNodeAck(BaseClusterService.java:1206)
	at org.elasticsearch.cluster.service.BaseClusterService$DelegetingAckListener.onNodeAck(BaseClusterService.java:1146)
	at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:1657)
	at org.elasticsearch.cluster.service.BaseClusterService.runTasks(BaseClusterService.java:583)
	at org.elasticsearch.cluster.service.BaseClusterService$ClusterServiceTaskBatcher.run(BaseClusterService.java:263)
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
	at org.elasticsearch.cluster.service.BaseClusterService$ClusterServiceTaskBatcher$UpdateTask.run(BaseClusterService.java:266)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2018-04-26 06:54:18,749 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:18,750 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:18,751 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:18,752 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:18,752 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:18,753 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 06:54:18,754 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError

@zllovesuki
Copy link
Author

zllovesuki commented Apr 26, 2018

(these are after I -X DELETE /twitter)

nodetool status:

root@elassandra-fmt01-0:/etc/cassandra# nodetool status
Datacenter: fmt01
=================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  65.49.34.153   482.89 KiB  256          100.0%            3a2236ad-bbe5-4d7a-9956-c425e09e91a8  01-15
UN  64.62.210.135  484.08 KiB  256          100.0%            b27dbc1f-7e23-4d12-acaf-75e41ad523a1  01-15
UN  64.71.160.168  593.72 KiB  256          100.0%            de73532a-94f6-45df-85fd-844270a3151a  01-15

nodetool gossipinfo:

root@elassandra-fmt01-0:/etc/cassandra# nodetool gossipinfo
/65.49.34.153
  generation:1524724359
  heartbeat:1919
  STATUS:62:NORMAL,-1014922541555553164
  LOAD:1907:499451.0
  SCHEMA:1746:98f1e96a-9afa-38ca-a7f7-df52eb21c831
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:172.16.0.101
  RPC_ADDRESS:3:65.49.34.153
  NET_VERSION:1:11
  HOST_ID:2:3a2236ad-bbe5-4d7a-9956-c425e09e91a8
  RPC_READY:77:true
  X1:1712:{"twitter":1}
  X2:1710:3a2236ad-bbe5-4d7a-9956-c425e09e91a8/8
  TOKENS:61:<hidden>
/64.62.210.135
  generation:1524724247
  heartbeat:2035
  STATUS:59:NORMAL,-104277563229732957
  LOAD:2031:500667.0
  SCHEMA:1860:98f1e96a-9afa-38ca-a7f7-df52eb21c831
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:172.16.0.102
  RPC_ADDRESS:3:64.62.210.135
  NET_VERSION:1:11
  HOST_ID:2:b27dbc1f-7e23-4d12-acaf-75e41ad523a1
  RPC_READY:73:true
  X1:1828:{"twitter":1}
  X2:1830:3a2236ad-bbe5-4d7a-9956-c425e09e91a8/8
  TOKENS:58:<hidden>
/64.71.160.168
  generation:1524724459
  heartbeat:1803
  STATUS:16:NORMAL,-1021721515604098786
  LOAD:1771:607968.0
  SCHEMA:1628:98f1e96a-9afa-38ca-a7f7-df52eb21c831
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:172.16.0.100
  RPC_ADDRESS:3:64.71.160.168
  NET_VERSION:1:11
  HOST_ID:2:de73532a-94f6-45df-85fd-844270a3151a
  RPC_READY:32:true
  X1:1599:{"twitter":1}
  X2:1601:3a2236ad-bbe5-4d7a-9956-c425e09e91a8/8
  TOKENS:15:<hidden>

_cluster/state?pretty:

{
  "cluster_name" : "Whirlpool",
  "version" : 74,
  "state_uuid" : "V1tg3cq_R7SKr5O0YQDJVA",
  "master_node" : "3a2236ad-bbe5-4d7a-9956-c425e09e91a8",
  "blocks" : { },
  "nodes" : {
    "3a2236ad-bbe5-4d7a-9956-c425e09e91a8" : {
      "name" : "172.16.0.101",
      "status" : "ALIVE",
      "ephemeral_id" : "3a2236ad-bbe5-4d7a-9956-c425e09e91a8",
      "transport_address" : "172.16.0.101:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "de73532a-94f6-45df-85fd-844270a3151a" : {
      "name" : "64.71.160.168",
      "status" : "ALIVE",
      "ephemeral_id" : "de73532a-94f6-45df-85fd-844270a3151a",
      "transport_address" : "64.71.160.168:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "b27dbc1f-7e23-4d12-acaf-75e41ad523a1" : {
      "name" : "64.62.210.135",
      "status" : "ALIVE",
      "ephemeral_id" : "b27dbc1f-7e23-4d12-acaf-75e41ad523a1",
      "transport_address" : "64.62.210.135:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 8,
    "cluster_uuid" : "3a2236ad-bbe5-4d7a-9956-c425e09e91a8",
    "templates" : { },
    "indices" : { },
    "index-graveyard" : {
      "tombstones" : [
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "TF9zOFiJSXWTjDK8Tf6tnQ"
          },
          "delete_date_in_millis" : 1524724789970
        },
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "Ox4WhQRST1Owbn4OnWITLQ"
          },
          "delete_date_in_millis" : 1524724847859
        },
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "KBgt4IAnQqKq9en9GiXG9w"
          },
          "delete_date_in_millis" : 1524724946291
        },
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "VVxdin1YQyS172wsv6MiNg"
          },
          "delete_date_in_millis" : 1524725917969
        }
      ]
    }
  },
  "routing_table" : {
    "indices" : { }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "de73532a-94f6-45df-85fd-844270a3151a" : [ ],
      "b27dbc1f-7e23-4d12-acaf-75e41ad523a1" : [ ],
      "3a2236ad-bbe5-4d7a-9956-c425e09e91a8" : [ ]
    }
  }
}

@zllovesuki
Copy link
Author

elasticsearch.yml has overwriting:

transport.bind_host: 0.0.0.0
transport.host: 172.16.0.100
transport.publish_host: 172.16.0.100

cassandra.yml has listen_address on private ip and broadcast_address on public ip

@vroyer
Copy link
Collaborator

vroyer commented Apr 26, 2018

Thanks' for your feed back.
I have released a patch for your issue in version 5.5.0.15 available at https://github.com/strapdata/elassandra5-rc/releases.
Here, you switch to IPv4 public IP.

Now you have created and deleted many times the index twitter, but for some reason, keyspace twitter is unavailable. Did you drop it ? Could you check that your schema is synchronized with a nodetool describecluster ?

@zllovesuki
Copy link
Author

zllovesuki commented Apr 26, 2018

UPDATED: see newer comment

Okay, so I installed the rc version, and with the following configuration, elassandra was able to start/bootstrap:
cassandra.yml:

listen_address: 172.16.0.100
broadcast_address: 64.71.160.168
listen_on_broadcast_address: true

elasticsearch.yml:

transport.bind_host: 0.0.0.0
transport.host: 172.16.0.100
transport.publish_host: 172.16.0.100

However, looking at /_cluster/state, it looks like they have a split-brain situation:
elassandra-fmt01-0:

{
  "cluster_name" : "Whirlpool",
  "version" : 45,
  "state_uuid" : "Pkcp4BJHQgOJn3xSL6b9Hg",
  "master_node" : "4232ff5e-7309-4b98-962e-ba676fd73d29",
  "blocks" : { },
  "nodes" : {
    "4232ff5e-7309-4b98-962e-ba676fd73d29" : {
      "name" : "172.16.0.100",
      "status" : "ALIVE",
      "ephemeral_id" : "4232ff5e-7309-4b98-962e-ba676fd73d29",
      "transport_address" : "172.16.0.100:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "d62292a2-3ec8-479b-b4f8-1363815d7bc4" : {
      "name" : "172.16.0.102",
      "status" : "ALIVE",
      "ephemeral_id" : "d62292a2-3ec8-479b-b4f8-1363815d7bc4",
      "transport_address" : "64.62.210.135:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "26ddecda-e017-4e7d-88ba-689e12ad63f2" : {
      "name" : "172.16.0.101",
      "status" : "ALIVE",
      "ephemeral_id" : "26ddecda-e017-4e7d-88ba-689e12ad63f2",
      "transport_address" : "65.49.34.153:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 4,
    "cluster_uuid" : "26ddecda-e017-4e7d-88ba-689e12ad63f2",
    "templates" : { },
    "indices" : { },
    "index-graveyard" : {
      "tombstones" : [
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "To2JoL-NQYGxVPX6-u8HtA"
          },
          "delete_date_in_millis" : 1524729616653
        },
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "wAlR20TCQHuHEOckfDLbnw"
          },
          "delete_date_in_millis" : 1524729724247
        }
      ]
    }
  },
  "routing_table" : {
    "indices" : { }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "4232ff5e-7309-4b98-962e-ba676fd73d29" : [ ],
      "d62292a2-3ec8-479b-b4f8-1363815d7bc4" : [ ],
      "26ddecda-e017-4e7d-88ba-689e12ad63f2" : [ ]
    }
  }
}

elassandra-fmt01-1:

{
  "cluster_name" : "Whirlpool",
  "version" : 38,
  "state_uuid" : "SzbuQo4JRaeM-CyA3Hm4OA",
  "master_node" : "d62292a2-3ec8-479b-b4f8-1363815d7bc4",
  "blocks" : { },
  "nodes" : {
    "26ddecda-e017-4e7d-88ba-689e12ad63f2" : {
      "name" : "172.16.0.101",
      "status" : "ALIVE",
      "ephemeral_id" : "26ddecda-e017-4e7d-88ba-689e12ad63f2",
      "transport_address" : "65.49.34.153:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "4232ff5e-7309-4b98-962e-ba676fd73d29" : {
      "name" : "64.71.160.168",
      "status" : "ALIVE",
      "ephemeral_id" : "4232ff5e-7309-4b98-962e-ba676fd73d29",
      "transport_address" : "64.71.160.168:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "d62292a2-3ec8-479b-b4f8-1363815d7bc4" : {
      "name" : "172.16.0.102",
      "status" : "ALIVE",
      "ephemeral_id" : "d62292a2-3ec8-479b-b4f8-1363815d7bc4",
      "transport_address" : "172.16.0.102:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 4,
    "cluster_uuid" : "26ddecda-e017-4e7d-88ba-689e12ad63f2",
    "templates" : { },
    "indices" : { },
    "index-graveyard" : {
      "tombstones" : [
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "To2JoL-NQYGxVPX6-u8HtA"
          },
          "delete_date_in_millis" : 1524729616653
        },
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "wAlR20TCQHuHEOckfDLbnw"
          },
          "delete_date_in_millis" : 1524729724247
        }
      ]
    }
  },
  "routing_table" : {
    "indices" : { }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "4232ff5e-7309-4b98-962e-ba676fd73d29" : [ ],
      "d62292a2-3ec8-479b-b4f8-1363815d7bc4" : [ ],
      "26ddecda-e017-4e7d-88ba-689e12ad63f2" : [ ]
    }
  }
}

elassandra-fmt01-2:

{
  "cluster_name" : "Whirlpool",
  "version" : 36,
  "state_uuid" : "PynrubXzR0iNuapaDuIRcA",
  "master_node" : "26ddecda-e017-4e7d-88ba-689e12ad63f2",
  "blocks" : { },
  "nodes" : {
    "d62292a2-3ec8-479b-b4f8-1363815d7bc4" : {
      "name" : "64.62.210.135",
      "status" : "ALIVE",
      "ephemeral_id" : "d62292a2-3ec8-479b-b4f8-1363815d7bc4",
      "transport_address" : "64.62.210.135:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "26ddecda-e017-4e7d-88ba-689e12ad63f2" : {
      "name" : "172.16.0.101",
      "status" : "ALIVE",
      "ephemeral_id" : "26ddecda-e017-4e7d-88ba-689e12ad63f2",
      "transport_address" : "172.16.0.101:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "4232ff5e-7309-4b98-962e-ba676fd73d29" : {
      "name" : "64.71.160.168",
      "status" : "ALIVE",
      "ephemeral_id" : "4232ff5e-7309-4b98-962e-ba676fd73d29",
      "transport_address" : "64.71.160.168:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 4,
    "cluster_uuid" : "26ddecda-e017-4e7d-88ba-689e12ad63f2",
    "templates" : { },
    "indices" : { },
    "index-graveyard" : {
      "tombstones" : [
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "To2JoL-NQYGxVPX6-u8HtA"
          },
          "delete_date_in_millis" : 1524729616653
        },
        {
          "index" : {
            "index_name" : "twitter",
            "index_uuid" : "wAlR20TCQHuHEOckfDLbnw"
          },
          "delete_date_in_millis" : 1524729724247
        }
      ]
    }
  },
  "routing_table" : {
    "indices" : { }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "4232ff5e-7309-4b98-962e-ba676fd73d29" : [ ],
      "d62292a2-3ec8-479b-b4f8-1363815d7bc4" : [ ],
      "26ddecda-e017-4e7d-88ba-689e12ad63f2" : [ ]
    }
  }
}

Notice how the versions are all different and the master node is not consistent.

So, using 5.5.0.15, how should I configure it so that internode communication is via private subnet but interdict is via public subnet?

@zllovesuki
Copy link
Author

zllovesuki commented Apr 26, 2018

UPDATED: see newer comment

nodetool describecluster yields:

root@elassandra-fmt01-0:/etc/cassandra# nodetool describecluster
Cluster Information:
	Name: Whirlpool
	Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
	DynamicEndPointSnitch: enabled
	Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
	Schema versions:
		8392629d-2255-3e65-8f04-e07eaa53803e: [65.49.34.153, 64.62.210.135, 64.71.160.168]

nodetool gossipinfo yields:

root@elassandra-fmt01-0:/etc/cassandra# nodetool gossipinfo
/65.49.34.153
  generation:1524729491
  heartbeat:514
  STATUS:62:NORMAL,-1060129253985440780
  LOAD:499:124617.0
  SCHEMA:303:8392629d-2255-3e65-8f04-e07eaa53803e
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:172.16.0.101
  RPC_ADDRESS:3:65.49.34.153
  NET_VERSION:1:11
  HOST_ID:2:26ddecda-e017-4e7d-88ba-689e12ad63f2
  RPC_READY:76:true
  X1:301:{"twitter":1}
  X2:299:26ddecda-e017-4e7d-88ba-689e12ad63f2/4
  TOKENS:61:<hidden>
/64.62.210.135
  generation:1524729394
  heartbeat:613
  STATUS:59:NORMAL,-106419404375351936
  LOAD:561:115889.0
  SCHEMA:399:8392629d-2255-3e65-8f04-e07eaa53803e
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:172.16.0.102
  RPC_ADDRESS:3:64.62.210.135
  NET_VERSION:1:11
  HOST_ID:2:d62292a2-3ec8-479b-b4f8-1363815d7bc4
  RPC_READY:73:true
  X1:402:{"twitter":1}
  X2:404:26ddecda-e017-4e7d-88ba-689e12ad63f2/4
  TOKENS:58:<hidden>
/64.71.160.168
  generation:1524729330
  heartbeat:685
  STATUS:52:NORMAL,-1254013397967417386
  LOAD:629:117529.0
  SCHEMA:471:8392629d-2255-3e65-8f04-e07eaa53803e
  DC:8:fmt01
  RACK:10:01-15
  RELEASE_VERSION:4:3.11.2
  INTERNAL_IP:6:172.16.0.100
  RPC_ADDRESS:3:64.71.160.168
  NET_VERSION:1:11
  HOST_ID:2:4232ff5e-7309-4b98-962e-ba676fd73d29
  RPC_READY:69:true
  X1:474:{"twitter":1}
  X2:476:26ddecda-e017-4e7d-88ba-689e12ad63f2/4
  TOKENS:51:<hidden>

(after -X DELETE /twitter)

root@elassandra-fmt01-0:/etc/cassandra# nodetool status elastic_admin
Datacenter: fmt01
=================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  65.49.34.153   121.7 KiB  256          100.0%            26ddecda-e017-4e7d-88ba-689e12ad63f2  01-15
UN  64.62.210.135  113.17 KiB  256          100.0%            d62292a2-3ec8-479b-b4f8-1363815d7bc4  01-15
UN  64.71.160.168  114.77 KiB  256          100.0%            4232ff5e-7309-4b98-962e-ba676fd73d29  01-15
root@elassandra-fmt01-0:/etc/cassandra# nodetool status twitter
Datacenter: fmt01
=================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
UN  65.49.34.153   121.7 KiB  256          35.9%             26ddecda-e017-4e7d-88ba-689e12ad63f2  01-15
UN  64.62.210.135  113.17 KiB  256          30.7%             d62292a2-3ec8-479b-b4f8-1363815d7bc4  01-15
UN  64.71.160.168  114.77 KiB  256          33.4%             4232ff5e-7309-4b98-962e-ba676fd73d29  01-15

@zllovesuki
Copy link
Author

zllovesuki commented Apr 26, 2018

UPDATED: This is the current configuration, please disregard the previous ones

I just tried this configuration:

cassandra.yml:

listen_address: 172.16.0.100
broadcast_address: 64.71.160.168
listen_on_broadcast_address: true
rpc_address: 0.0.0.0
broadcast_rpc_address: 172.16.0.100

elasticsearch.yml:

transport.bind_host: 172.16.0.100
transport.host: 172.16.0.100
transport.publish_host: 172.16.0.100
network.bind_host: 0.0.0.0
network.host: 172.16.0.100
network.publish_host: 172.16.0.100

These are the /_cluster/state info:

root@elassandra-fmt01-0:

{
  "cluster_name" : "Whirlpool",
  "version" : 17,
  "state_uuid" : "WcSSSfalS9WDFaFU7xzy4g",
  "master_node" : "ff77e33c-5b11-4338-a356-101375cf21c3",
  "blocks" : { },
  "nodes" : {
    "6a06914c-137e-4e58-8d61-dcfde8a4ad8c" : {
      "name" : "172.16.0.101",
      "status" : "ALIVE",
      "ephemeral_id" : "6a06914c-137e-4e58-8d61-dcfde8a4ad8c",
      "transport_address" : "172.16.0.101:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "d74e1af9-3987-4819-acdd-064bf20cfa2d" : {
      "name" : "172.16.0.102",
      "status" : "ALIVE",
      "ephemeral_id" : "d74e1af9-3987-4819-acdd-064bf20cfa2d",
      "transport_address" : "172.16.0.102:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "ff77e33c-5b11-4338-a356-101375cf21c3" : {
      "name" : "172.16.0.100",
      "status" : "ALIVE",
      "ephemeral_id" : "ff77e33c-5b11-4338-a356-101375cf21c3",
      "transport_address" : "172.16.0.100:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 0,
    "cluster_uuid" : "ff77e33c-5b11-4338-a356-101375cf21c3",
    "templates" : { },
    "indices" : { },
    "index-graveyard" : {
      "tombstones" : [ ]
    }
  },
  "routing_table" : {
    "indices" : { }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "ff77e33c-5b11-4338-a356-101375cf21c3" : [ ],
      "6a06914c-137e-4e58-8d61-dcfde8a4ad8c" : [ ],
      "d74e1af9-3987-4819-acdd-064bf20cfa2d" : [ ]
    }
  }
}

root@elassandra-fmt01-2:

{
  "cluster_name" : "Whirlpool",
  "version" : 10,
  "state_uuid" : "-YlbzcQJQVWm8Hix1RtUyg",
  "master_node" : "6a06914c-137e-4e58-8d61-dcfde8a4ad8c",
  "blocks" : { },
  "nodes" : {
    "ff77e33c-5b11-4338-a356-101375cf21c3" : {
      "name" : "64.71.160.168",
      "status" : "ALIVE",
      "ephemeral_id" : "ff77e33c-5b11-4338-a356-101375cf21c3",
      "transport_address" : "172.16.0.100:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "d74e1af9-3987-4819-acdd-064bf20cfa2d" : {
      "name" : "64.62.210.135",
      "status" : "ALIVE",
      "ephemeral_id" : "d74e1af9-3987-4819-acdd-064bf20cfa2d",
      "transport_address" : "172.16.0.102:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "6a06914c-137e-4e58-8d61-dcfde8a4ad8c" : {
      "name" : "172.16.0.101",
      "status" : "ALIVE",
      "ephemeral_id" : "6a06914c-137e-4e58-8d61-dcfde8a4ad8c",
      "transport_address" : "172.16.0.101:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 0,
    "cluster_uuid" : "ff77e33c-5b11-4338-a356-101375cf21c3",
    "templates" : { },
    "indices" : { },
    "index-graveyard" : {
      "tombstones" : [ ]
    }
  },
  "routing_table" : {
    "indices" : { }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "ff77e33c-5b11-4338-a356-101375cf21c3" : [ ],
      "6a06914c-137e-4e58-8d61-dcfde8a4ad8c" : [ ],
      "d74e1af9-3987-4819-acdd-064bf20cfa2d" : [ ]
    }
  }
}

elassandra-fmt01-2:

{
  "cluster_name" : "Whirlpool",
  "version" : 14,
  "state_uuid" : "HZcGeaOOT2CazNu-R2iIUg",
  "master_node" : "d74e1af9-3987-4819-acdd-064bf20cfa2d",
  "blocks" : { },
  "nodes" : {
    "ff77e33c-5b11-4338-a356-101375cf21c3" : {
      "name" : "64.71.160.168",
      "status" : "ALIVE",
      "ephemeral_id" : "ff77e33c-5b11-4338-a356-101375cf21c3",
      "transport_address" : "172.16.0.100:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "6a06914c-137e-4e58-8d61-dcfde8a4ad8c" : {
      "name" : "172.16.0.101",
      "status" : "ALIVE",
      "ephemeral_id" : "6a06914c-137e-4e58-8d61-dcfde8a4ad8c",
      "transport_address" : "172.16.0.101:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "d74e1af9-3987-4819-acdd-064bf20cfa2d" : {
      "name" : "172.16.0.102",
      "status" : "ALIVE",
      "ephemeral_id" : "d74e1af9-3987-4819-acdd-064bf20cfa2d",
      "transport_address" : "172.16.0.102:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 0,
    "cluster_uuid" : "ff77e33c-5b11-4338-a356-101375cf21c3",
    "templates" : { },
    "indices" : { },
    "index-graveyard" : {
      "tombstones" : [ ]
    }
  },
  "routing_table" : {
    "indices" : { }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "ff77e33c-5b11-4338-a356-101375cf21c3" : [ ],
      "6a06914c-137e-4e58-8d61-dcfde8a4ad8c" : [ ],
      "d74e1af9-3987-4819-acdd-064bf20cfa2d" : [ ]
    }
  }
}

Now the transport address is thru private subnet but the names are all different (also the versions don't match).

@zllovesuki
Copy link
Author

zllovesuki commented Apr 26, 2018

Not working:

$ curl -X PUT 'http://64.71.160.168:9200/twitter/doc/1?pretty' -H 'Content-Type: application/json' -d '
{
    "user": "Poulpy",
    "post_date": "2017/10/4 13:12:00",
    "message": "Elassandra adds dynamic mapping to Cassandra"
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index",
        "index_uuid" : "C4Z-WH_6QciSwhfHRuCz7Q",
        "index" : "twitter"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index",
    "index_uuid" : "C4Z-WH_6QciSwhfHRuCz7Q",
    "index" : "twitter"
  },
  "status" : 404
}
2018-04-26 08:26:44,368 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] StorageService.java:3846 setLoggingLevel set log level to TRACE for classes under 'index.search.slowlog.query.twitter' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
2018-04-26 08:26:44,369 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] StorageService.java:3846 setLoggingLevel set log level to TRACE for classes under 'index.search.slowlog.fetch.twitter' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
2018-04-26 08:26:44,447 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] MigrationManager.java:331 announceNewKeyspace Create new Keyspace: KeyspaceMetadata{name=twitter, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, fmt01=1}}, tables=[], views=[], functions=[], types=[]}
2018-04-26 08:26:44,502 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:44,509 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] CassandraDiscovery.java:526 isNormal Node name=[172.16.0.101] address=[/172.16.0.101] state not found
2018-04-26 08:26:44,509 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] CassandraDiscovery.java:526 isNormal Node name=[172.16.0.102] address=[/172.16.0.102] state not found
2018-04-26 08:26:44,552 INFO  [metadataSchemaUpdater] MigrationManager.java:457 announceColumnFamilyUpdate Update table 'elastic_admin/metadata' From org.apache.cassandra.config.CFMetaData@3bb75753[cfId=e1fbf4c0-492a-11e8-b11d-73dc54d8358f,ksName=elastic_admin,cfName=metadata,flags=[COMPOUND],params=TableParams{comment={"meta-data":{"version":0,"cluster_uuid":"ff77e33c-5b11-4338-a356-101375cf21c3","templates":{},"index-graveyard":{"tombstones":[]}}}, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={max_threshold=32, min_threshold=4}}, compression=org.apache.cassandra.schema.CompressionParams@7d0c27cf, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [metadata owner version]],partitionKeyColumns=[cluster_name],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[owner, version, cluster_name, metadata],droppedColumns={},triggers=[],indexes=[]] To org.apache.cassandra.config.CFMetaData@1a05dabb[cfId=e1fbf4c0-492a-11e8-b11d-73dc54d8358f,ksName=elastic_admin,cfName=metadata,flags=[COMPOUND],params=TableParams{comment={"meta-data":{"version":1,"cluster_uuid":"ff77e33c-5b11-4338-a356-101375cf21c3","templates":{},"indices":{"twitter":{"version":1,"state":"open","settings":{"index.creation_date":"1524731204332","index.provided_name":"twitter","index.uuid":"C4Z-WH_6QciSwhfHRuCz7Q","index.version.created":"5050099"},"mappings":[],"aliases":{}}},"index-graveyard":{"tombstones":[]}}}, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={max_threshold=32, min_threshold=4}}, compression=org.apache.cassandra.schema.CompressionParams@7d0c27cf, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [metadata owner version]],partitionKeyColumns=[cluster_name],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[owner, version, cluster_name, metadata],droppedColumns={},triggers=[],indexes=[]]
2018-04-26 08:26:44,555 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:44,556 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] StorageService.java:3846 setLoggingLevel set log level to TRACE for classes under 'index.search.slowlog.query.twitter' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
2018-04-26 08:26:44,556 INFO  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] StorageService.java:3846 setLoggingLevel set log level to TRACE for classes under 'index.search.slowlog.fetch.twitter' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE')
2018-04-26 08:26:44,602 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:44,756 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:44,758 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:44,760 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:46,182 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:46,184 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:46,439 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:26:46,442 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] IndexRoutingTable.java:380 <init> Keyspace not available: java.lang.AssertionError
2018-04-26 08:27:14,514 WARN  [elasticsearch[172.16.0.100][generic][T#2]] TransportReplicationAction.java:859 finishWithUnexpectedFailure unexpected error during the primary phase for action [indices:data/write/bulk[s]], request [BulkShardRequest [[twitter][0]] containing [index {[twitter][doc][1], source[
{
    "user": "Poulpy",
    "post_date": "2017/10/4 13:12:00",
    "message": "Elassandra adds dynamic mapping to Cassandra"
}]}]]
org.elasticsearch.index.IndexNotFoundException: no such index
	at org.elasticsearch.cluster.routing.RoutingTable.shardRoutingTable(RoutingTable.java:141)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.primary(TransportReplicationAction.java:738)
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:667)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:147)
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:92)
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170)
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84)
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:363)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:478)
	at org.elasticsearch.action.bulk.TransportBulkAction$1.onResponse(TransportBulkAction.java:179)
	at org.elasticsearch.action.bulk.TransportBulkAction$1.onResponse(TransportBulkAction.java:175)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:84)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onResponse(TransportMasterNodeAction.java:149)
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.onResponse(TransportMasterNodeAction.java:146)
	at org.elasticsearch.action.admin.indices.create.TransportCreateIndexAction.lambda$masterOperation$0(TransportCreateIndexAction.java:82)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59)
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.lambda$createIndex$1(MetaDataCreateIndexService.java:217)
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
	at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAckTimeout(AckedClusterStateUpdateTask.java:79)
	at org.elasticsearch.cluster.service.BaseClusterService$SafeAckedClusterStateTaskListener.onAckTimeout(BaseClusterService.java:1017)
	at org.elasticsearch.cluster.service.BaseClusterService$AckCountDownListener.onTimeout(BaseClusterService.java:1214)
	at org.elasticsearch.cluster.service.BaseClusterService$AckCountDownListener.lambda$new$0(BaseClusterService.java:1182)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

/_cluster/state now has an additional metadata block (no routing table):

  "metadata" : {
    "version" : 1,
    "cluster_uuid" : "ff77e33c-5b11-4338-a356-101375cf21c3",
    "templates" : { },
    "indices" : {
      "twitter" : {
        "state" : "open",
        "settings" : {
          "index" : {
            "creation_date" : "1524731204332",
            "number_of_shards" : "3",
            "number_of_replicas" : "0",
            "uuid" : "C4Z-WH_6QciSwhfHRuCz7Q",
            "version" : {
              "created" : "5050099"
            },
            "provided_name" : "twitter"
          }
        },
        "mappings" : { },
        "aliases" : [ ],
        "primary_terms" : {
          "0" : 0,
          "1" : 0,
          "2" : 0
        },
        "in_sync_allocations" : {
          "2" : [ ],
          "1" : [ ],
          "0" : [ ]
        }
      }
    },
    "index-graveyard" : {
      "tombstones" : [ ]
    }
  },

@vroyer
Copy link
Collaborator

vroyer commented Apr 26, 2018 via email

@zllovesuki
Copy link
Author

root@elassandra-fmt01-0:~# cqlsh -u cassandra -p cassandra
Connected to Whirlpool at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh> describe keyspaces;

twitter      system         system_distributed  system_schema
system_auth  elastic_admin  system_traces     

cassandra@cqlsh> describe keyspace twitter

CREATE KEYSPACE twitter WITH replication = {'class': 'NetworkTopologyStrategy', 'fmt01': '1'}  AND durable_writes = true;

Same on all nodes. Although, class=org.apache.cassandra.locator.NetworkTopologyStrategy, fmt01=1 looks very suspicious (because it only does replication factor of 1, shouldn't it be 3?)

@zllovesuki
Copy link
Author

zllovesuki commented Apr 26, 2018

Also, noticed that it did mention state not found:

2018-04-26 08:26:44,509 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] CassandraDiscovery.java:526 isNormal Node name=[172.16.0.101] address=[/172.16.0.101] state not found
2018-04-26 08:26:44,509 WARN  [elasticsearch[172.16.0.100][clusterService#updateTask][T#1]] CassandraDiscovery.java:526 isNormal Node name=[172.16.0.102] address=[/172.16.0.102] state not found

@zllovesuki zllovesuki changed the title ClusterService.java:1542 publishAndApplyChanges Cassandra issue: java.lang.NullPointerException: null Elasticsearch refuses to work when listen_address differs from broadcast_address Apr 26, 2018
@zllovesuki
Copy link
Author

I've managed to get a working configuration.

cassandra.yml:

listen_address: 172.16.0.100
broadcast_address: 64.71.160.168
listen_on_broadcast_address: true
rpc_address: 64.71.160.168

elasticsearch.yml:

transport.bind_host: 0.0.0.0
network.bind_host: 0.0.0.0

As such, it was able to create the index (but apparently one of my test node is too slow, 30s timeout to acknowledge mapping update. However running -X PUT /twitter again worked):

{
  "cluster_name" : "Whirlpool",
  "version" : 28,
  "state_uuid" : "gEYIMTjqRDG8-bSEWxNSOA",
  "master_node" : "e021f7ed-1f37-4664-93ef-19fa517fd779",
  "blocks" : { },
  "nodes" : {
    "e021f7ed-1f37-4664-93ef-19fa517fd779" : {
      "name" : "172.16.0.100",
      "status" : "ALIVE",
      "ephemeral_id" : "e021f7ed-1f37-4664-93ef-19fa517fd779",
      "transport_address" : "64.71.160.168:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "bd4d83e3-ffc2-464d-bd2d-ceedef621a48" : {
      "name" : "172.16.0.102",
      "status" : "ALIVE",
      "ephemeral_id" : "bd4d83e3-ffc2-464d-bd2d-ceedef621a48",
      "transport_address" : "64.62.210.135:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    },
    "1757e1f2-d030-4993-9fbe-c7140b2ad44e" : {
      "name" : "172.16.0.101",
      "status" : "ALIVE",
      "ephemeral_id" : "1757e1f2-d030-4993-9fbe-c7140b2ad44e",
      "transport_address" : "65.49.34.153:9300",
      "attributes" : {
        "rack" : "01-15",
        "dc" : "fmt01"
      }
    }
  },
  "metadata" : {
    "version" : 2,
    "cluster_uuid" : "e021f7ed-1f37-4664-93ef-19fa517fd779",
    "templates" : { },
    "indices" : {
      "twitter" : {
        "state" : "open",
        "settings" : {
          "index" : {
            "creation_date" : "1524764807515",
            "number_of_shards" : "3",
            "number_of_replicas" : "0",
            "uuid" : "7NYtYoZWSKuat5iTP4AxJA",
            "version" : {
              "created" : "5050099"
            },
            "provided_name" : "twitter"
          }
        },
        "mappings" : {
          "doc" : {
            "properties" : {
              "post_date" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "message" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "user" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              }
            }
          }
        },
        "aliases" : [ ],
        "primary_terms" : {
          "0" : 0,
          "1" : 0,
          "2" : 0
        },
        "in_sync_allocations" : {
          "2" : [ ],
          "1" : [ ],
          "0" : [ ]
        }
      }
    },
    "index-graveyard" : {
      "tombstones" : [ ]
    }
  },
  "routing_table" : {
    "indices" : {
      "twitter" : {
        "shards" : {
          "2" : [
            {
              "state" : "STARTED",
              "primary" : true,
              "node" : "bd4d83e3-ffc2-464d-bd2d-ceedef621a48",
              "relocating_node" : null,
              "shard" : 2,
              "index" : "twitter",
              "token_ranges" : [
                "(-9223372036854775808,-9207086341881892102]",
                "(-9177408036589429549,-9166571365741182949]",
                "(-9142188689193659142,-9131052293804415574]",
                "(-9081992317971635865,-9064354844827859123]",
                "(-9010082164002282234,-8959790950853826596]",
                "(-8747402907428044753,-8659962981164020924]",
                "(-8534378643155817005,-8476970071976715352]",
                "(-8370385968807417027,-8326899406984078229]",
                "(-8249930013654206648,-8229571107802391895]",
                "(-8159639752176211088,-8108648962295640965]",
                "(-8104022435150587630,-8054150916104868379]",
                "(-7963795854286289525,-7869662108434949727]",
                "(-7831750182861378409,-7808766566388753017]",
                "(-7690320537435035876,-7671079588872329462]",
                "(-7595984959857869586,-7575436918663566328]",
                "(-7456236465670562636,-7416658275139145547]",
                "(-7367810317337273670,-7357068419041019024]",
                "(-7223354085793474218,-7221833145402021933]",
                "(-7195579756599031635,-7190382470578228179]",
                "(-7116737196362517558,-7097090186347625928]",
                "(-7011629332919682845,-6974586810098076832]",
                "(-6887983664048201388,-6819125215298941683]",
                "(-6762903679170685435,-6673995218730104953]",
                "(-6632618939854799169,-6601766677695704020]",
                "(-6537491279263248869,-6497027502615282075]",
                "(-6240145912203391126,-6159979341627687699]",
                "(-6144201193668365204,-6129773694540726598]",
                "(-6048112955166336675,-5953315743345673121]",
                "(-5845766449907293878,-5808040912809516467]",
                "(-5802315625531302436,-5797818401291407433]",
                "(-5796663731228752595,-5673810889237868136]",
                "(-5669747315005495104,-5656480134983638187]",
                "(-5630803517394195877,-5612844165608471744]",
                "(-5468887289203126131,-5394969667317476133]",
                "(-5369233000769603473,-5366525025299931917]",
                "(-5246976024276330270,-5202938309589817093]",
                "(-5156351645442935046,-5153889773198331620]",
                "(-5076496057144057412,-5012357193262868226]",
                "(-5003842590471224586,-4973720276109385005]",
                "(-4951477777191422790,-4898374902918569417]",
                "(-4897524569307419531,-4813889769145348044]",
                "(-4659758510266697608,-4658049660921352174]",
                "(-4616741655289378946,-4610279716045450236]",
                "(-4595216317508028075,-4575101049129623695]",
                "(-4557570391222458228,-4524524591592425476]",
                "(-4492232085090802116,-4482875623926168520]",
                "(-4415633098923482876,-4328911087459118924]",
                "(-4312345754744489445,-4287520010218115072]",
                "(-4227827473047437845,-4131827904794520726]",
                "(-4041733271906863320,-4033240344484853737]",
                "(-3927694206320414218,-3827715225643418743]",
                "(-3768847227245779074,-3672097928730824439]",
                "(-3666170970834564631,-3645827960859965395]",
                "(-3550247132202179844,-3536601442880139274]",
                "(-3510979463042602347,-3388018628590158562]",
                "(-3380356296853850128,-3374864003332373000]",
                "(-3348787255161868136,-3345882581412775033]",
                "(-3252467741669561779,-3227915924181072149]",
                "(-3221480063259969743,-3199597925155476437]",
                "(-2818353703369652005,-2798514195557508189]",
                "(-2604169715936767733,-2593139335897315408]",
                "(-2551993695304834557,-2540753783130835677]",
                "(-2528434822622323961,-2494050463747041050]",
                "(-2463476759470186981,-2408471301010528834]",
                "(-2404397642761195259,-2399482403998910913]",
                "(-2278037377177831331,-2235887577229324610]",
                "(-2200812816853781602,-2178028334953437283]",
                "(-2138210537263516289,-2127449923477522378]",
                "(-2037121425922561620,-2006919530062687927]",
                "(-1838768795644616052,-1798560013695157675]",
                "(-1723906978721212483,-1719497320615655250]",
                "(-1695198905292483267,-1675697634940281031]",
                "(-1519325803733028714,-1497940248681987877]",
                "(-1444146252963247882,-1426897630512617093]",
                "(-1426453719318613349,-1363261904920081294]",
                "(-1170881599226209153,-1156628728578635129]",
                "(-1106619542875767496,-1088807060680842621]",
                "(-1084222888468883138,-1040586676284875100]",
                "(-1021391077830646260,-999514634602845483]",
                "(-957512136607530220,-868467015972771318]",
                "(-864696458996931109,-832613742359152203]",
                "(-758650111360148124,-737710204117823315]",
                "(-649578017011185273,-639294090230264304]",
                "(-419552039552503814,-363273312509395668]",
                "(-315598617291245777,-287321996454838061]",
                "(-196167153032116189,-139285086313993600]",
                "(-120006214026553231,42503443349475570]",
                "(89694158702341033,150761618339097974]",
                "(392131304865516294,395338209398788922]",
                "(418730396410804594,435620466292158003]",
                "(566601480057436041,589486145243112399]",
                "(764100695175992337,819697516728481417]",
                "(893112022595021384,909277857360868331]",
                "(1175470480998293994,1177123869022269314]",
                "(1331353277012468356,1372872776242136026]",
                "(1444702200961172634,1568575502085927093]",
                "(1689565155668118741,1743086562335528638]",
                "(1865179185954824289,1938647031174059126]",
                "(1998498752108054070,2004887856210806873]",
                "(2021869720372204244,2063545244238224010]",
                "(2127820829828387348,2170253078339436748]",
                "(2233622322681138171,2261128905714084746]",
                "(2261526937758341643,2291726491184178877]",
                "(2605171324708643517,2649618329017469140]",
                "(2680116080176327274,2711164487656644640]",
                "(2745991490441588335,2781187690737954502]",
                "(2783838757402233436,2845205369679645850]",
                "(2867998726118861053,2897523676889778161]",
                "(2942077496403075533,2969851407537735475]",
                "(3001195184239726073,3013316343159239729]",
                "(3032840490469964329,3051469837388269590]",
                "(3163960818202640362,3174031453968003175]",
                "(3221296403323129007,3368330241204414603]",
                "(3386205286883450222,3388464125960242047]",
                "(3411021830331935421,3476055586412806338]",
                "(3505800927197764530,3510595188313377084]",
                "(3529604762963489193,3554455245591637203]",
                "(3587479208450392457,3598250134222825303]",
                "(3808066427895328936,3824756698990053684]",
                "(3871335070474566952,3888247251802588183]",
                "(3908319470500628797,3911508202917431888]",
                "(3984124253005822204,4066465522494888665]",
                "(4132413434046740428,4253327597206362081]",
                "(4283249979692848581,4288598812309801272]",
                "(4363473361597907381,4379359632456542587]",
                "(4429269402060193031,4430865227839341821]",
                "(4546403471009864002,4546852998847864152]",
                "(4554028945988839367,4609851810228305236]",
                "(4797059259873273348,4902730862557356587]",
                "(5005300018878581367,5043113337897651400]",
                "(5207696626562339674,5226870210783718617]",
                "(5262597744340127542,5362238487658613068]",
                "(5379497455182707044,5434370467725853492]",
                "(5620130792283031186,5623951157255038536]",
                "(5640656681195611496,5646113358273505290]",
                "(5657280808675754737,5662026247123171918]",
                "(5667627655611186181,5689628098834015208]",
                "(5695438821189119688,5730095482939842697]",
                "(5916973408178703080,6050611292385734389]",
                "(6079154228273183729,6090089609473339695]",
                "(6233102408608229003,6238887659671392930]",
                "(6268397881596606388,6283167365273706631]",
                "(6382424778322053954,6405352309981402563]",
                "(6425709650188168169,6494205821215136348]",
                "(6599388239037707951,6724195608022104060]",
                "(6877147673659306574,6925189137023589324]",
                "(7025662255343686174,7055807262727319424]",
                "(7099440082474576491,7114406049240854360]",
                "(7181079371809651303,7199576419099781804]",
                "(7214675372183956393,7232733341982721609]",
                "(7292775917038263233,7309980407472880887]",
                "(7351650085248126195,7359030485367457448]",
                "(7380227913635029036,7414154199197577355]",
                "(7547556318670138169,7570094251113682739]",
                "(7649427894846954898,7651810029329144495]",
                "(7667541804338737129,7710964422865265962]",
                "(7928030646662068947,8089838211260662478]",
                "(8105963935427010978,8206628778459195202]",
                "(8440992966323396669,8452098743862133946]",
                "(8486663669237009036,8490080457235737117]",
                "(8584200698150181853,8659664415469601414]",
                "(8701561361806505978,8771341666294407497]",
                "(8890621759510705487,8970419178269923673]",
                "(9073493434922174946,9116475822629024863]",
                "(9122063044812619144,9125478626552811469]",
                "(9183410537987850115,9223372036854775807]"
              ],
              "allocation_id" : {
                "id" : "dummy_alloc_id"
              }
            }
          ],
          "1" : [
            {
              "state" : "STARTED",
              "primary" : true,
              "node" : "1757e1f2-d030-4993-9fbe-c7140b2ad44e",
              "relocating_node" : null,
              "shard" : 1,
              "index" : "twitter",
              "token_ranges" : [
                "(-9207086341881892102,-9177408036589429549]",
                "(-9155062754416793332,-9142188689193659142]",
                "(-9097018060661304853,-9081992317971635865]",
                "(-9064354844827859123,-9036544862476112590]",
                "(-9016393532985771561,-9010082164002282234]",
                "(-8844109249410262049,-8802154410391062931]",
                "(-8581535908872670604,-8534378643155817005]",
                "(-8476970071976715352,-8429371768858528205]",
                "(-8422416658547653948,-8370385968807417027]",
                "(-8281769774098899526,-8249930013654206648]",
                "(-8229571107802391895,-8211317276646145105]",
                "(-8209081938956272120,-8193281572689785280]",
                "(-8054150916104868379,-7988452850685196768]",
                "(-7966560463151722847,-7963795854286289525]",
                "(-7808766566388753017,-7690320537435035876]",
                "(-7667671969947696932,-7595984959857869586]",
                "(-7503227943535988347,-7456236465670562636]",
                "(-7416658275139145547,-7367810317337273670]",
                "(-7357068419041019024,-7324196752962989215]",
                "(-7221833145402021933,-7195579756599031635]",
                "(-7190382470578228179,-7116737196362517558]",
                "(-7069157637206069896,-7046772127308864953]",
                "(-7025681645228772950,-7011629332919682845]",
                "(-6973212370289148578,-6887983664048201388]",
                "(-6819125215298941683,-6762903679170685435]",
                "(-6497027502615282075,-6391664159682937929]",
                "(-6309970683227739100,-6282650458357129015]",
                "(-6159979341627687699,-6147472926555487003]",
                "(-6129773694540726598,-6048112955166336675]",
                "(-5903493697561410463,-5845766449907293878]",
                "(-5808040912809516467,-5802315625531302436]",
                "(-5656480134983638187,-5640052345966669815]",
                "(-5612844165608471744,-5603906737471235858]",
                "(-5596266843792813273,-5499878335748008712]",
                "(-5394969667317476133,-5369233000769603473]",
                "(-5352938732032416512,-5306240455252423848]",
                "(-5187370831263236566,-5156351645442935046]",
                "(-5136867707683631017,-5076496057144057412]",
                "(-5012357193262868226,-5003842590471224586]",
                "(-4898374902918569417,-4897524569307419531]",
                "(-4813889769145348044,-4799513522902022248]",
                "(-4658049660921352174,-4623404095866352088]",
                "(-4482875623926168520,-4452153834743874415]",
                "(-4328911087459118924,-4312345754744489445]",
                "(-4287520010218115072,-4269183957453519562]",
                "(-4238887874094615725,-4227827473047437845]",
                "(-4116785939434107733,-4105642353877318698]",
                "(-4033240344484853737,-4029723412717967207]",
                "(-4024723320988703468,-3927694206320414218]",
                "(-3827715225643418743,-3768847227245779074]",
                "(-3672097928730824439,-3666170970834564631]",
                "(-3536601442880139274,-3510979463042602347]",
                "(-3315636442862463298,-3252467741669561779]",
                "(-3227915924181072149,-3221480063259969743]",
                "(-3199192254203282437,-3054389300490465931]",
                "(-3028012974862091887,-3014829888702105676]",
                "(-2894648424351835905,-2865630853339592949]",
                "(-2850148989560661235,-2823355389487034944]",
                "(-2798514195557508189,-2673446354987498053]",
                "(-2632923042355434765,-2604169715936767733]",
                "(-2593139335897315408,-2590161891830978339]",
                "(-2585175156406586560,-2551993695304834557]",
                "(-2469027288286104886,-2463476759470186981]",
                "(-2408471301010528834,-2407695150877320662]",
                "(-2346886022351771856,-2303918959149563860]",
                "(-2235887577229324610,-2235406711466365121]",
                "(-2141581132387137819,-2138210537263516289]",
                "(-2093726558791956445,-2037121425922561620]",
                "(-1938398471771138452,-1845912392692404926]",
                "(-1736940720452291590,-1723906978721212483]",
                "(-1719497320615655250,-1695198905292483267]",
                "(-1675697634940281031,-1519325803733028714]",
                "(-1456678842704243053,-1444146252963247882]",
                "(-1320773696990158853,-1292783265289362175]",
                "(-1251418718669374597,-1248269873519971999]",
                "(-1181868650856552100,-1170881599226209153]",
                "(-1156628728578635129,-1155089599132410278]",
                "(-1141429489583158553,-1120388934560947486]",
                "(-1088807060680842621,-1084222888468883138]",
                "(-1040586676284875100,-1021391077830646260]",
                "(-999514634602845483,-995453724119491669]",
                "(-980840263871901082,-957512136607530220]",
                "(-832613742359152203,-814589200046464419]",
                "(-737710204117823315,-649578017011185273]",
                "(-639294090230264304,-621840291887196954]",
                "(-517790684309078071,-419552039552503814]",
                "(-348090541756488517,-315598617291245777]",
                "(-285566612576288574,-267542250101988677]",
                "(42503443349475570,83091460727950403]",
                "(150761618339097974,164822415516337189]",
                "(227398291462125966,392131304865516294]",
                "(395338209398788922,395859520398417380]",
                "(435620466292158003,493450805444386166]",
                "(601546818874379285,612215652435822332]",
                "(627188611878215442,651102451639373615]",
                "(679465782284251391,707932382443299488]",
                "(734684173768167592,744475082786849430]",
                "(909277857360868331,1175470480998293994]",
                "(1208586032241662051,1262556497087715200]",
                "(1292746028051866165,1331353277012468356]",
                "(1372872776242136026,1413194292717986667]",
                "(1597329577788859457,1689565155668118741]",
                "(1766355061290536111,1843765767282046745]",
                "(1938647031174059126,1996010148283751797]",
                "(2004887856210806873,2021869720372204244]",
                "(2170253078339436748,2233622322681138171]",
                "(2261128905714084746,2261526937758341643]",
                "(2364946079926566075,2368892554747482374]",
                "(2526520843378449759,2542422759095283090]",
                "(2567018585474147204,2602587955661315881]",
                "(2742030639361065098,2745991490441588335]",
                "(2781187690737954502,2783838757402233436]",
                "(2845205369679645850,2867998726118861053]",
                "(2969851407537735475,3001195184239726073]",
                "(3017795737263471904,3027448326167725313]",
                "(3051469837388269590,3131156370221092581]",
                "(3377601333037178063,3386205286883450222]",
                "(3388464125960242047,3395856014177234141]",
                "(3476055586412806338,3505800927197764530]",
                "(3510595188313377084,3529604762963489193]",
                "(3562313775583509919,3587479208450392457]",
                "(3824756698990053684,3826569017806138739]",
                "(3853857434377248814,3871335070474566952]",
                "(3911508202917431888,3934143792528955777]",
                "(4066465522494888665,4069678577776741774]",
                "(4253327597206362081,4283249979692848581]",
                "(4288598812309801272,4363473361597907381]",
                "(4379359632456542587,4386070882514524042]",
                "(4392017264417534076,4429269402060193031]",
                "(4533320853118693073,4546403471009864002]",
                "(4609851810228305236,4614975173775179130]",
                "(4783267355924094706,4794354707345505008]",
                "(4902730862557356587,4907946037800886153]",
                "(4983812323026902822,4995636847191205675]",
                "(4998715014208768157,5005300018878581367]",
                "(5146450558326891157,5207696626562339674]",
                "(5362238487658613068,5379497455182707044]",
                "(5434370467725853492,5478731165379239536]",
                "(5511934326296457059,5551269097062000812]",
                "(5689628098834015208,5695438821189119688]",
                "(5837513789075859462,5887623930446068243]",
                "(6050611292385734389,6065033779686224154]",
                "(6073393275312728268,6079154228273183729]",
                "(6090089609473339695,6233102408608229003]",
                "(6238887659671392930,6240736112961270935]",
                "(6247469316077790988,6268397881596606388]",
                "(6283167365273706631,6351494374054270965]",
                "(6494205821215136348,6498326540339215012]",
                "(6569021582430029962,6597959520125951486]",
                "(6724195608022104060,6808612008420511655]",
                "(6841090605435365371,6877147673659306574]",
                "(6925189137023589324,7025662255343686174]",
                "(7055807262727319424,7099440082474576491]",
                "(7199576419099781804,7214675372183956393]",
                "(7232733341982721609,7290876542154642517]",
                "(7323439137352426358,7351650085248126195]",
                "(7359030485367457448,7366278703596010815]",
                "(7477607398105624145,7501476478632678984]",
                "(7502496064295543264,7547556318670138169]",
                "(7570094251113682739,7576017857519664486]",
                "(7615626023489389371,7649427894846954898]",
                "(7710964422865265962,7719995121343477129]",
                "(7767680971353441264,7924718468916853777]",
                "(8089838211260662478,8105963935427010978]",
                "(8206628778459195202,8230860910278958828]",
                "(8238840301073800072,8264032002299557817]",
                "(8336735911190597193,8440992966323396669]",
                "(8457150334853223236,8470478769021375990]",
                "(8659664415469601414,8666841686873417869]",
                "(8771341666294407497,8783637386283340932]",
                "(9062684181299674460,9065885268000025271]",
                "(9131143268228968383,9133625023306928982]",
                "(9151577294789513038,9174699050991224568]"
              ],
              "allocation_id" : {
                "id" : "dummy_alloc_id"
              }
            }
          ],
          "0" : [
            {
              "state" : "STARTED",
              "primary" : true,
              "node" : "e021f7ed-1f37-4664-93ef-19fa517fd779",
              "relocating_node" : null,
              "shard" : 0,
              "index" : "twitter",
              "token_ranges" : [
                "(-9166571365741182949,-9155062754416793332]",
                "(-9131052293804415574,-9097018060661304853]",
                "(-9036544862476112590,-9016393532985771561]",
                "(-8959790950853826596,-8844109249410262049]",
                "(-8802154410391062931,-8747402907428044753]",
                "(-8659962981164020924,-8581535908872670604]",
                "(-8429371768858528205,-8422416658547653948]",
                "(-8326899406984078229,-8281769774098899526]",
                "(-8211317276646145105,-8209081938956272120]",
                "(-8193281572689785280,-8159639752176211088]",
                "(-8108648962295640965,-8104022435150587630]",
                "(-7988452850685196768,-7966560463151722847]",
                "(-7869662108434949727,-7831750182861378409]",
                "(-7671079588872329462,-7667671969947696932]",
                "(-7575436918663566328,-7503227943535988347]",
                "(-7324196752962989215,-7223354085793474218]",
                "(-7097090186347625928,-7069157637206069896]",
                "(-7046772127308864953,-7025681645228772950]",
                "(-6974586810098076832,-6973212370289148578]",
                "(-6673995218730104953,-6632618939854799169]",
                "(-6601766677695704020,-6537491279263248869]",
                "(-6391664159682937929,-6309970683227739100]",
                "(-6282650458357129015,-6240145912203391126]",
                "(-6147472926555487003,-6144201193668365204]",
                "(-5953315743345673121,-5903493697561410463]",
                "(-5797818401291407433,-5796663731228752595]",
                "(-5673810889237868136,-5669747315005495104]",
                "(-5640052345966669815,-5630803517394195877]",
                "(-5603906737471235858,-5596266843792813273]",
                "(-5499878335748008712,-5468887289203126131]",
                "(-5366525025299931917,-5352938732032416512]",
                "(-5306240455252423848,-5246976024276330270]",
                "(-5202938309589817093,-5187370831263236566]",
                "(-5153889773198331620,-5136867707683631017]",
                "(-4973720276109385005,-4951477777191422790]",
                "(-4799513522902022248,-4659758510266697608]",
                "(-4623404095866352088,-4616741655289378946]",
                "(-4610279716045450236,-4595216317508028075]",
                "(-4575101049129623695,-4557570391222458228]",
                "(-4524524591592425476,-4492232085090802116]",
                "(-4452153834743874415,-4415633098923482876]",
                "(-4269183957453519562,-4238887874094615725]",
                "(-4131827904794520726,-4116785939434107733]",
                "(-4105642353877318698,-4041733271906863320]",
                "(-4029723412717967207,-4024723320988703468]",
                "(-3645827960859965395,-3550247132202179844]",
                "(-3388018628590158562,-3380356296853850128]",
                "(-3374864003332373000,-3348787255161868136]",
                "(-3345882581412775033,-3315636442862463298]",
                "(-3199597925155476437,-3199192254203282437]",
                "(-3054389300490465931,-3028012974862091887]",
                "(-3014829888702105676,-2894648424351835905]",
                "(-2865630853339592949,-2850148989560661235]",
                "(-2823355389487034944,-2818353703369652005]",
                "(-2673446354987498053,-2632923042355434765]",
                "(-2590161891830978339,-2585175156406586560]",
                "(-2540753783130835677,-2528434822622323961]",
                "(-2494050463747041050,-2469027288286104886]",
                "(-2407695150877320662,-2404397642761195259]",
                "(-2399482403998910913,-2346886022351771856]",
                "(-2303918959149563860,-2278037377177831331]",
                "(-2235406711466365121,-2200812816853781602]",
                "(-2178028334953437283,-2141581132387137819]",
                "(-2127449923477522378,-2093726558791956445]",
                "(-2006919530062687927,-1938398471771138452]",
                "(-1845912392692404926,-1838768795644616052]",
                "(-1798560013695157675,-1736940720452291590]",
                "(-1497940248681987877,-1456678842704243053]",
                "(-1426897630512617093,-1426453719318613349]",
                "(-1363261904920081294,-1320773696990158853]",
                "(-1292783265289362175,-1251418718669374597]",
                "(-1248269873519971999,-1181868650856552100]",
                "(-1155089599132410278,-1141429489583158553]",
                "(-1120388934560947486,-1106619542875767496]",
                "(-995453724119491669,-980840263871901082]",
                "(-868467015972771318,-864696458996931109]",
                "(-814589200046464419,-758650111360148124]",
                "(-621840291887196954,-517790684309078071]",
                "(-363273312509395668,-348090541756488517]",
                "(-287321996454838061,-285566612576288574]",
                "(-267542250101988677,-196167153032116189]",
                "(-139285086313993600,-120006214026553231]",
                "(83091460727950403,89694158702341033]",
                "(164822415516337189,227398291462125966]",
                "(395859520398417380,418730396410804594]",
                "(493450805444386166,566601480057436041]",
                "(589486145243112399,601546818874379285]",
                "(612215652435822332,627188611878215442]",
                "(651102451639373615,679465782284251391]",
                "(707932382443299488,734684173768167592]",
                "(744475082786849430,764100695175992337]",
                "(819697516728481417,893112022595021384]",
                "(1177123869022269314,1208586032241662051]",
                "(1262556497087715200,1292746028051866165]",
                "(1413194292717986667,1444702200961172634]",
                "(1568575502085927093,1597329577788859457]",
                "(1743086562335528638,1766355061290536111]",
                "(1843765767282046745,1865179185954824289]",
                "(1996010148283751797,1998498752108054070]",
                "(2063545244238224010,2127820829828387348]",
                "(2291726491184178877,2364946079926566075]",
                "(2368892554747482374,2526520843378449759]",
                "(2542422759095283090,2567018585474147204]",
                "(2602587955661315881,2605171324708643517]",
                "(2649618329017469140,2680116080176327274]",
                "(2711164487656644640,2742030639361065098]",
                "(2897523676889778161,2942077496403075533]",
                "(3013316343159239729,3017795737263471904]",
                "(3027448326167725313,3032840490469964329]",
                "(3131156370221092581,3163960818202640362]",
                "(3174031453968003175,3221296403323129007]",
                "(3368330241204414603,3377601333037178063]",
                "(3395856014177234141,3411021830331935421]",
                "(3554455245591637203,3562313775583509919]",
                "(3598250134222825303,3808066427895328936]",
                "(3826569017806138739,3853857434377248814]",
                "(3888247251802588183,3908319470500628797]",
                "(3934143792528955777,3984124253005822204]",
                "(4069678577776741774,4132413434046740428]",
                "(4386070882514524042,4392017264417534076]",
                "(4430865227839341821,4533320853118693073]",
                "(4546852998847864152,4554028945988839367]",
                "(4614975173775179130,4783267355924094706]",
                "(4794354707345505008,4797059259873273348]",
                "(4907946037800886153,4983812323026902822]",
                "(4995636847191205675,4998715014208768157]",
                "(5043113337897651400,5146450558326891157]",
                "(5226870210783718617,5262597744340127542]",
                "(5478731165379239536,5511934326296457059]",
                "(5551269097062000812,5620130792283031186]",
                "(5623951157255038536,5640656681195611496]",
                "(5646113358273505290,5657280808675754737]",
                "(5662026247123171918,5667627655611186181]",
                "(5730095482939842697,5837513789075859462]",
                "(5887623930446068243,5916973408178703080]",
                "(6065033779686224154,6073393275312728268]",
                "(6240736112961270935,6247469316077790988]",
                "(6351494374054270965,6382424778322053954]",
                "(6405352309981402563,6425709650188168169]",
                "(6498326540339215012,6569021582430029962]",
                "(6597959520125951486,6599388239037707951]",
                "(6808612008420511655,6841090605435365371]",
                "(7114406049240854360,7181079371809651303]",
                "(7290876542154642517,7292775917038263233]",
                "(7309980407472880887,7323439137352426358]",
                "(7366278703596010815,7380227913635029036]",
                "(7414154199197577355,7477607398105624145]",
                "(7501476478632678984,7502496064295543264]",
                "(7576017857519664486,7615626023489389371]",
                "(7651810029329144495,7667541804338737129]",
                "(7719995121343477129,7767680971353441264]",
                "(7924718468916853777,7928030646662068947]",
                "(8230860910278958828,8238840301073800072]",
                "(8264032002299557817,8336735911190597193]",
                "(8452098743862133946,8457150334853223236]",
                "(8470478769021375990,8486663669237009036]",
                "(8490080457235737117,8584200698150181853]",
                "(8666841686873417869,8701561361806505978]",
                "(8783637386283340932,8890621759510705487]",
                "(8970419178269923673,9062684181299674460]",
                "(9065885268000025271,9073493434922174946]",
                "(9116475822629024863,9122063044812619144]",
                "(9125478626552811469,9131143268228968383]",
                "(9133625023306928982,9151577294789513038]",
                "(9174699050991224568,9183410537987850115]"
              ],
              "allocation_id" : {
                "id" : "dummy_alloc_id"
              }
            }
          ]
        }
      }
    }
  },
  "routing_nodes" : {
    "unassigned" : [ ],
    "nodes" : {
      "1757e1f2-d030-4993-9fbe-c7140b2ad44e" : [
        {
          "state" : "STARTED",
          "primary" : true,
          "node" : "1757e1f2-d030-4993-9fbe-c7140b2ad44e",
          "relocating_node" : null,
          "shard" : 1,
          "index" : "twitter",
          "token_ranges" : [
            "(-9207086341881892102,-9177408036589429549]",
            "(-9155062754416793332,-9142188689193659142]",
            "(-9097018060661304853,-9081992317971635865]",
            "(-9064354844827859123,-9036544862476112590]",
            "(-9016393532985771561,-9010082164002282234]",
            "(-8844109249410262049,-8802154410391062931]",
            "(-8581535908872670604,-8534378643155817005]",
            "(-8476970071976715352,-8429371768858528205]",
            "(-8422416658547653948,-8370385968807417027]",
            "(-8281769774098899526,-8249930013654206648]",
            "(-8229571107802391895,-8211317276646145105]",
            "(-8209081938956272120,-8193281572689785280]",
            "(-8054150916104868379,-7988452850685196768]",
            "(-7966560463151722847,-7963795854286289525]",
            "(-7808766566388753017,-7690320537435035876]",
            "(-7667671969947696932,-7595984959857869586]",
            "(-7503227943535988347,-7456236465670562636]",
            "(-7416658275139145547,-7367810317337273670]",
            "(-7357068419041019024,-7324196752962989215]",
            "(-7221833145402021933,-7195579756599031635]",
            "(-7190382470578228179,-7116737196362517558]",
            "(-7069157637206069896,-7046772127308864953]",
            "(-7025681645228772950,-7011629332919682845]",
            "(-6973212370289148578,-6887983664048201388]",
            "(-6819125215298941683,-6762903679170685435]",
            "(-6497027502615282075,-6391664159682937929]",
            "(-6309970683227739100,-6282650458357129015]",
            "(-6159979341627687699,-6147472926555487003]",
            "(-6129773694540726598,-6048112955166336675]",
            "(-5903493697561410463,-5845766449907293878]",
            "(-5808040912809516467,-5802315625531302436]",
            "(-5656480134983638187,-5640052345966669815]",
            "(-5612844165608471744,-5603906737471235858]",
            "(-5596266843792813273,-5499878335748008712]",
            "(-5394969667317476133,-5369233000769603473]",
            "(-5352938732032416512,-5306240455252423848]",
            "(-5187370831263236566,-5156351645442935046]",
            "(-5136867707683631017,-5076496057144057412]",
            "(-5012357193262868226,-5003842590471224586]",
            "(-4898374902918569417,-4897524569307419531]",
            "(-4813889769145348044,-4799513522902022248]",
            "(-4658049660921352174,-4623404095866352088]",
            "(-4482875623926168520,-4452153834743874415]",
            "(-4328911087459118924,-4312345754744489445]",
            "(-4287520010218115072,-4269183957453519562]",
            "(-4238887874094615725,-4227827473047437845]",
            "(-4116785939434107733,-4105642353877318698]",
            "(-4033240344484853737,-4029723412717967207]",
            "(-4024723320988703468,-3927694206320414218]",
            "(-3827715225643418743,-3768847227245779074]",
            "(-3672097928730824439,-3666170970834564631]",
            "(-3536601442880139274,-3510979463042602347]",
            "(-3315636442862463298,-3252467741669561779]",
            "(-3227915924181072149,-3221480063259969743]",
            "(-3199192254203282437,-3054389300490465931]",
            "(-3028012974862091887,-3014829888702105676]",
            "(-2894648424351835905,-2865630853339592949]",
            "(-2850148989560661235,-2823355389487034944]",
            "(-2798514195557508189,-2673446354987498053]",
            "(-2632923042355434765,-2604169715936767733]",
            "(-2593139335897315408,-2590161891830978339]",
            "(-2585175156406586560,-2551993695304834557]",
            "(-2469027288286104886,-2463476759470186981]",
            "(-2408471301010528834,-2407695150877320662]",
            "(-2346886022351771856,-2303918959149563860]",
            "(-2235887577229324610,-2235406711466365121]",
            "(-2141581132387137819,-2138210537263516289]",
            "(-2093726558791956445,-2037121425922561620]",
            "(-1938398471771138452,-1845912392692404926]",
            "(-1736940720452291590,-1723906978721212483]",
            "(-1719497320615655250,-1695198905292483267]",
            "(-1675697634940281031,-1519325803733028714]",
            "(-1456678842704243053,-1444146252963247882]",
            "(-1320773696990158853,-1292783265289362175]",
            "(-1251418718669374597,-1248269873519971999]",
            "(-1181868650856552100,-1170881599226209153]",
            "(-1156628728578635129,-1155089599132410278]",
            "(-1141429489583158553,-1120388934560947486]",
            "(-1088807060680842621,-1084222888468883138]",
            "(-1040586676284875100,-1021391077830646260]",
            "(-999514634602845483,-995453724119491669]",
            "(-980840263871901082,-957512136607530220]",
            "(-832613742359152203,-814589200046464419]",
            "(-737710204117823315,-649578017011185273]",
            "(-639294090230264304,-621840291887196954]",
            "(-517790684309078071,-419552039552503814]",
            "(-348090541756488517,-315598617291245777]",
            "(-285566612576288574,-267542250101988677]",
            "(42503443349475570,83091460727950403]",
            "(150761618339097974,164822415516337189]",
            "(227398291462125966,392131304865516294]",
            "(395338209398788922,395859520398417380]",
            "(435620466292158003,493450805444386166]",
            "(601546818874379285,612215652435822332]",
            "(627188611878215442,651102451639373615]",
            "(679465782284251391,707932382443299488]",
            "(734684173768167592,744475082786849430]",
            "(909277857360868331,1175470480998293994]",
            "(1208586032241662051,1262556497087715200]",
            "(1292746028051866165,1331353277012468356]",
            "(1372872776242136026,1413194292717986667]",
            "(1597329577788859457,1689565155668118741]",
            "(1766355061290536111,1843765767282046745]",
            "(1938647031174059126,1996010148283751797]",
            "(2004887856210806873,2021869720372204244]",
            "(2170253078339436748,2233622322681138171]",
            "(2261128905714084746,2261526937758341643]",
            "(2364946079926566075,2368892554747482374]",
            "(2526520843378449759,2542422759095283090]",
            "(2567018585474147204,2602587955661315881]",
            "(2742030639361065098,2745991490441588335]",
            "(2781187690737954502,2783838757402233436]",
            "(2845205369679645850,2867998726118861053]",
            "(2969851407537735475,3001195184239726073]",
            "(3017795737263471904,3027448326167725313]",
            "(3051469837388269590,3131156370221092581]",
            "(3377601333037178063,3386205286883450222]",
            "(3388464125960242047,3395856014177234141]",
            "(3476055586412806338,3505800927197764530]",
            "(3510595188313377084,3529604762963489193]",
            "(3562313775583509919,3587479208450392457]",
            "(3824756698990053684,3826569017806138739]",
            "(3853857434377248814,3871335070474566952]",
            "(3911508202917431888,3934143792528955777]",
            "(4066465522494888665,4069678577776741774]",
            "(4253327597206362081,4283249979692848581]",
            "(4288598812309801272,4363473361597907381]",
            "(4379359632456542587,4386070882514524042]",
            "(4392017264417534076,4429269402060193031]",
            "(4533320853118693073,4546403471009864002]",
            "(4609851810228305236,4614975173775179130]",
            "(4783267355924094706,4794354707345505008]",
            "(4902730862557356587,4907946037800886153]",
            "(4983812323026902822,4995636847191205675]",
            "(4998715014208768157,5005300018878581367]",
            "(5146450558326891157,5207696626562339674]",
            "(5362238487658613068,5379497455182707044]",
            "(5434370467725853492,5478731165379239536]",
            "(5511934326296457059,5551269097062000812]",
            "(5689628098834015208,5695438821189119688]",
            "(5837513789075859462,5887623930446068243]",
            "(6050611292385734389,6065033779686224154]",
            "(6073393275312728268,6079154228273183729]",
            "(6090089609473339695,6233102408608229003]",
            "(6238887659671392930,6240736112961270935]",
            "(6247469316077790988,6268397881596606388]",
            "(6283167365273706631,6351494374054270965]",
            "(6494205821215136348,6498326540339215012]",
            "(6569021582430029962,6597959520125951486]",
            "(6724195608022104060,6808612008420511655]",
            "(6841090605435365371,6877147673659306574]",
            "(6925189137023589324,7025662255343686174]",
            "(7055807262727319424,7099440082474576491]",
            "(7199576419099781804,7214675372183956393]",
            "(7232733341982721609,7290876542154642517]",
            "(7323439137352426358,7351650085248126195]",
            "(7359030485367457448,7366278703596010815]",
            "(7477607398105624145,7501476478632678984]",
            "(7502496064295543264,7547556318670138169]",
            "(7570094251113682739,7576017857519664486]",
            "(7615626023489389371,7649427894846954898]",
            "(7710964422865265962,7719995121343477129]",
            "(7767680971353441264,7924718468916853777]",
            "(8089838211260662478,8105963935427010978]",
            "(8206628778459195202,8230860910278958828]",
            "(8238840301073800072,8264032002299557817]",
            "(8336735911190597193,8440992966323396669]",
            "(8457150334853223236,8470478769021375990]",
            "(8659664415469601414,8666841686873417869]",
            "(8771341666294407497,8783637386283340932]",
            "(9062684181299674460,9065885268000025271]",
            "(9131143268228968383,9133625023306928982]",
            "(9151577294789513038,9174699050991224568]"
          ],
          "allocation_id" : {
            "id" : "dummy_alloc_id"
          }
        }
      ],
      "e021f7ed-1f37-4664-93ef-19fa517fd779" : [
        {
          "state" : "STARTED",
          "primary" : true,
          "node" : "e021f7ed-1f37-4664-93ef-19fa517fd779",
          "relocating_node" : null,
          "shard" : 0,
          "index" : "twitter",
          "token_ranges" : [
            "(-9166571365741182949,-9155062754416793332]",
            "(-9131052293804415574,-9097018060661304853]",
            "(-9036544862476112590,-9016393532985771561]",
            "(-8959790950853826596,-8844109249410262049]",
            "(-8802154410391062931,-8747402907428044753]",
            "(-8659962981164020924,-8581535908872670604]",
            "(-8429371768858528205,-8422416658547653948]",
            "(-8326899406984078229,-8281769774098899526]",
            "(-8211317276646145105,-8209081938956272120]",
            "(-8193281572689785280,-8159639752176211088]",
            "(-8108648962295640965,-8104022435150587630]",
            "(-7988452850685196768,-7966560463151722847]",
            "(-7869662108434949727,-7831750182861378409]",
            "(-7671079588872329462,-7667671969947696932]",
            "(-7575436918663566328,-7503227943535988347]",
            "(-7324196752962989215,-7223354085793474218]",
            "(-7097090186347625928,-7069157637206069896]",
            "(-7046772127308864953,-7025681645228772950]",
            "(-6974586810098076832,-6973212370289148578]",
            "(-6673995218730104953,-6632618939854799169]",
            "(-6601766677695704020,-6537491279263248869]",
            "(-6391664159682937929,-6309970683227739100]",
            "(-6282650458357129015,-6240145912203391126]",
            "(-6147472926555487003,-6144201193668365204]",
            "(-5953315743345673121,-5903493697561410463]",
            "(-5797818401291407433,-5796663731228752595]",
            "(-5673810889237868136,-5669747315005495104]",
            "(-5640052345966669815,-5630803517394195877]",
            "(-5603906737471235858,-5596266843792813273]",
            "(-5499878335748008712,-5468887289203126131]",
            "(-5366525025299931917,-5352938732032416512]",
            "(-5306240455252423848,-5246976024276330270]",
            "(-5202938309589817093,-5187370831263236566]",
            "(-5153889773198331620,-5136867707683631017]",
            "(-4973720276109385005,-4951477777191422790]",
            "(-4799513522902022248,-4659758510266697608]",
            "(-4623404095866352088,-4616741655289378946]",
            "(-4610279716045450236,-4595216317508028075]",
            "(-4575101049129623695,-4557570391222458228]",
            "(-4524524591592425476,-4492232085090802116]",
            "(-4452153834743874415,-4415633098923482876]",
            "(-4269183957453519562,-4238887874094615725]",
            "(-4131827904794520726,-4116785939434107733]",
            "(-4105642353877318698,-4041733271906863320]",
            "(-4029723412717967207,-4024723320988703468]",
            "(-3645827960859965395,-3550247132202179844]",
            "(-3388018628590158562,-3380356296853850128]",
            "(-3374864003332373000,-3348787255161868136]",
            "(-3345882581412775033,-3315636442862463298]",
            "(-3199597925155476437,-3199192254203282437]",
            "(-3054389300490465931,-3028012974862091887]",
            "(-3014829888702105676,-2894648424351835905]",
            "(-2865630853339592949,-2850148989560661235]",
            "(-2823355389487034944,-2818353703369652005]",
            "(-2673446354987498053,-2632923042355434765]",
            "(-2590161891830978339,-2585175156406586560]",
            "(-2540753783130835677,-2528434822622323961]",
            "(-2494050463747041050,-2469027288286104886]",
            "(-2407695150877320662,-2404397642761195259]",
            "(-2399482403998910913,-2346886022351771856]",
            "(-2303918959149563860,-2278037377177831331]",
            "(-2235406711466365121,-2200812816853781602]",
            "(-2178028334953437283,-2141581132387137819]",
            "(-2127449923477522378,-2093726558791956445]",
            "(-2006919530062687927,-1938398471771138452]",
            "(-1845912392692404926,-1838768795644616052]",
            "(-1798560013695157675,-1736940720452291590]",
            "(-1497940248681987877,-1456678842704243053]",
            "(-1426897630512617093,-1426453719318613349]",
            "(-1363261904920081294,-1320773696990158853]",
            "(-1292783265289362175,-1251418718669374597]",
            "(-1248269873519971999,-1181868650856552100]",
            "(-1155089599132410278,-1141429489583158553]",
            "(-1120388934560947486,-1106619542875767496]",
            "(-995453724119491669,-980840263871901082]",
            "(-868467015972771318,-864696458996931109]",
            "(-814589200046464419,-758650111360148124]",
            "(-621840291887196954,-517790684309078071]",
            "(-363273312509395668,-348090541756488517]",
            "(-287321996454838061,-285566612576288574]",
            "(-267542250101988677,-196167153032116189]",
            "(-139285086313993600,-120006214026553231]",
            "(83091460727950403,89694158702341033]",
            "(164822415516337189,227398291462125966]",
            "(395859520398417380,418730396410804594]",
            "(493450805444386166,566601480057436041]",
            "(589486145243112399,601546818874379285]",
            "(612215652435822332,627188611878215442]",
            "(651102451639373615,679465782284251391]",
            "(707932382443299488,734684173768167592]",
            "(744475082786849430,764100695175992337]",
            "(819697516728481417,893112022595021384]",
            "(1177123869022269314,1208586032241662051]",
            "(1262556497087715200,1292746028051866165]",
            "(1413194292717986667,1444702200961172634]",
            "(1568575502085927093,1597329577788859457]",
            "(1743086562335528638,1766355061290536111]",
            "(1843765767282046745,1865179185954824289]",
            "(1996010148283751797,1998498752108054070]",
            "(2063545244238224010,2127820829828387348]",
            "(2291726491184178877,2364946079926566075]",
            "(2368892554747482374,2526520843378449759]",
            "(2542422759095283090,2567018585474147204]",
            "(2602587955661315881,2605171324708643517]",
            "(2649618329017469140,2680116080176327274]",
            "(2711164487656644640,2742030639361065098]",
            "(2897523676889778161,2942077496403075533]",
            "(3013316343159239729,3017795737263471904]",
            "(3027448326167725313,3032840490469964329]",
            "(3131156370221092581,3163960818202640362]",
            "(3174031453968003175,3221296403323129007]",
            "(3368330241204414603,3377601333037178063]",
            "(3395856014177234141,3411021830331935421]",
            "(3554455245591637203,3562313775583509919]",
            "(3598250134222825303,3808066427895328936]",
            "(3826569017806138739,3853857434377248814]",
            "(3888247251802588183,3908319470500628797]",
            "(3934143792528955777,3984124253005822204]",
            "(4069678577776741774,4132413434046740428]",
            "(4386070882514524042,4392017264417534076]",
            "(4430865227839341821,4533320853118693073]",
            "(4546852998847864152,4554028945988839367]",
            "(4614975173775179130,4783267355924094706]",
            "(4794354707345505008,4797059259873273348]",
            "(4907946037800886153,4983812323026902822]",
            "(4995636847191205675,4998715014208768157]",
            "(5043113337897651400,5146450558326891157]",
            "(5226870210783718617,5262597744340127542]",
            "(5478731165379239536,5511934326296457059]",
            "(5551269097062000812,5620130792283031186]",
            "(5623951157255038536,5640656681195611496]",
            "(5646113358273505290,5657280808675754737]",
            "(5662026247123171918,5667627655611186181]",
            "(5730095482939842697,5837513789075859462]",
            "(5887623930446068243,5916973408178703080]",
            "(6065033779686224154,6073393275312728268]",
            "(6240736112961270935,6247469316077790988]",
            "(6351494374054270965,6382424778322053954]",
            "(6405352309981402563,6425709650188168169]",
            "(6498326540339215012,6569021582430029962]",
            "(6597959520125951486,6599388239037707951]",
            "(6808612008420511655,6841090605435365371]",
            "(7114406049240854360,7181079371809651303]",
            "(7290876542154642517,7292775917038263233]",
            "(7309980407472880887,7323439137352426358]",
            "(7366278703596010815,7380227913635029036]",
            "(7414154199197577355,7477607398105624145]",
            "(7501476478632678984,7502496064295543264]",
            "(7576017857519664486,7615626023489389371]",
            "(7651810029329144495,7667541804338737129]",
            "(7719995121343477129,7767680971353441264]",
            "(7924718468916853777,7928030646662068947]",
            "(8230860910278958828,8238840301073800072]",
            "(8264032002299557817,8336735911190597193]",
            "(8452098743862133946,8457150334853223236]",
            "(8470478769021375990,8486663669237009036]",
            "(8490080457235737117,8584200698150181853]",
            "(8666841686873417869,8701561361806505978]",
            "(8783637386283340932,8890621759510705487]",
            "(8970419178269923673,9062684181299674460]",
            "(9065885268000025271,9073493434922174946]",
            "(9116475822629024863,9122063044812619144]",
            "(9125478626552811469,9131143268228968383]",
            "(9133625023306928982,9151577294789513038]",
            "(9174699050991224568,9183410537987850115]"
          ],
          "allocation_id" : {
            "id" : "dummy_alloc_id"
          }
        }
      ],
      "bd4d83e3-ffc2-464d-bd2d-ceedef621a48" : [
        {
          "state" : "STARTED",
          "primary" : true,
          "node" : "bd4d83e3-ffc2-464d-bd2d-ceedef621a48",
          "relocating_node" : null,
          "shard" : 2,
          "index" : "twitter",
          "token_ranges" : [
            "(-9223372036854775808,-9207086341881892102]",
            "(-9177408036589429549,-9166571365741182949]",
            "(-9142188689193659142,-9131052293804415574]",
            "(-9081992317971635865,-9064354844827859123]",
            "(-9010082164002282234,-8959790950853826596]",
            "(-8747402907428044753,-8659962981164020924]",
            "(-8534378643155817005,-8476970071976715352]",
            "(-8370385968807417027,-8326899406984078229]",
            "(-8249930013654206648,-8229571107802391895]",
            "(-8159639752176211088,-8108648962295640965]",
            "(-8104022435150587630,-8054150916104868379]",
            "(-7963795854286289525,-7869662108434949727]",
            "(-7831750182861378409,-7808766566388753017]",
            "(-7690320537435035876,-7671079588872329462]",
            "(-7595984959857869586,-7575436918663566328]",
            "(-7456236465670562636,-7416658275139145547]",
            "(-7367810317337273670,-7357068419041019024]",
            "(-7223354085793474218,-7221833145402021933]",
            "(-7195579756599031635,-7190382470578228179]",
            "(-7116737196362517558,-7097090186347625928]",
            "(-7011629332919682845,-6974586810098076832]",
            "(-6887983664048201388,-6819125215298941683]",
            "(-6762903679170685435,-6673995218730104953]",
            "(-6632618939854799169,-6601766677695704020]",
            "(-6537491279263248869,-6497027502615282075]",
            "(-6240145912203391126,-6159979341627687699]",
            "(-6144201193668365204,-6129773694540726598]",
            "(-6048112955166336675,-5953315743345673121]",
            "(-5845766449907293878,-5808040912809516467]",
            "(-5802315625531302436,-5797818401291407433]",
            "(-5796663731228752595,-5673810889237868136]",
            "(-5669747315005495104,-5656480134983638187]",
            "(-5630803517394195877,-5612844165608471744]",
            "(-5468887289203126131,-5394969667317476133]",
            "(-5369233000769603473,-5366525025299931917]",
            "(-5246976024276330270,-5202938309589817093]",
            "(-5156351645442935046,-5153889773198331620]",
            "(-5076496057144057412,-5012357193262868226]",
            "(-5003842590471224586,-4973720276109385005]",
            "(-4951477777191422790,-4898374902918569417]",
            "(-4897524569307419531,-4813889769145348044]",
            "(-4659758510266697608,-4658049660921352174]",
            "(-4616741655289378946,-4610279716045450236]",
            "(-4595216317508028075,-4575101049129623695]",
            "(-4557570391222458228,-4524524591592425476]",
            "(-4492232085090802116,-4482875623926168520]",
            "(-4415633098923482876,-4328911087459118924]",
            "(-4312345754744489445,-4287520010218115072]",
            "(-4227827473047437845,-4131827904794520726]",
            "(-4041733271906863320,-4033240344484853737]",
            "(-3927694206320414218,-3827715225643418743]",
            "(-3768847227245779074,-3672097928730824439]",
            "(-3666170970834564631,-3645827960859965395]",
            "(-3550247132202179844,-3536601442880139274]",
            "(-3510979463042602347,-3388018628590158562]",
            "(-3380356296853850128,-3374864003332373000]",
            "(-3348787255161868136,-3345882581412775033]",
            "(-3252467741669561779,-3227915924181072149]",
            "(-3221480063259969743,-3199597925155476437]",
            "(-2818353703369652005,-2798514195557508189]",
            "(-2604169715936767733,-2593139335897315408]",
            "(-2551993695304834557,-2540753783130835677]",
            "(-2528434822622323961,-2494050463747041050]",
            "(-2463476759470186981,-2408471301010528834]",
            "(-2404397642761195259,-2399482403998910913]",
            "(-2278037377177831331,-2235887577229324610]",
            "(-2200812816853781602,-2178028334953437283]",
            "(-2138210537263516289,-2127449923477522378]",
            "(-2037121425922561620,-2006919530062687927]",
            "(-1838768795644616052,-1798560013695157675]",
            "(-1723906978721212483,-1719497320615655250]",
            "(-1695198905292483267,-1675697634940281031]",
            "(-1519325803733028714,-1497940248681987877]",
            "(-1444146252963247882,-1426897630512617093]",
            "(-1426453719318613349,-1363261904920081294]",
            "(-1170881599226209153,-1156628728578635129]",
            "(-1106619542875767496,-1088807060680842621]",
            "(-1084222888468883138,-1040586676284875100]",
            "(-1021391077830646260,-999514634602845483]",
            "(-957512136607530220,-868467015972771318]",
            "(-864696458996931109,-832613742359152203]",
            "(-758650111360148124,-737710204117823315]",
            "(-649578017011185273,-639294090230264304]",
            "(-419552039552503814,-363273312509395668]",
            "(-315598617291245777,-287321996454838061]",
            "(-196167153032116189,-139285086313993600]",
            "(-120006214026553231,42503443349475570]",
            "(89694158702341033,150761618339097974]",
            "(392131304865516294,395338209398788922]",
            "(418730396410804594,435620466292158003]",
            "(566601480057436041,589486145243112399]",
            "(764100695175992337,819697516728481417]",
            "(893112022595021384,909277857360868331]",
            "(1175470480998293994,1177123869022269314]",
            "(1331353277012468356,1372872776242136026]",
            "(1444702200961172634,1568575502085927093]",
            "(1689565155668118741,1743086562335528638]",
            "(1865179185954824289,1938647031174059126]",
            "(1998498752108054070,2004887856210806873]",
            "(2021869720372204244,2063545244238224010]",
            "(2127820829828387348,2170253078339436748]",
            "(2233622322681138171,2261128905714084746]",
            "(2261526937758341643,2291726491184178877]",
            "(2605171324708643517,2649618329017469140]",
            "(2680116080176327274,2711164487656644640]",
            "(2745991490441588335,2781187690737954502]",
            "(2783838757402233436,2845205369679645850]",
            "(2867998726118861053,2897523676889778161]",
            "(2942077496403075533,2969851407537735475]",
            "(3001195184239726073,3013316343159239729]",
            "(3032840490469964329,3051469837388269590]",
            "(3163960818202640362,3174031453968003175]",
            "(3221296403323129007,3368330241204414603]",
            "(3386205286883450222,3388464125960242047]",
            "(3411021830331935421,3476055586412806338]",
            "(3505800927197764530,3510595188313377084]",
            "(3529604762963489193,3554455245591637203]",
            "(3587479208450392457,3598250134222825303]",
            "(3808066427895328936,3824756698990053684]",
            "(3871335070474566952,3888247251802588183]",
            "(3908319470500628797,3911508202917431888]",
            "(3984124253005822204,4066465522494888665]",
            "(4132413434046740428,4253327597206362081]",
            "(4283249979692848581,4288598812309801272]",
            "(4363473361597907381,4379359632456542587]",
            "(4429269402060193031,4430865227839341821]",
            "(4546403471009864002,4546852998847864152]",
            "(4554028945988839367,4609851810228305236]",
            "(4797059259873273348,4902730862557356587]",
            "(5005300018878581367,5043113337897651400]",
            "(5207696626562339674,5226870210783718617]",
            "(5262597744340127542,5362238487658613068]",
            "(5379497455182707044,5434370467725853492]",
            "(5620130792283031186,5623951157255038536]",
            "(5640656681195611496,5646113358273505290]",
            "(5657280808675754737,5662026247123171918]",
            "(5667627655611186181,5689628098834015208]",
            "(5695438821189119688,5730095482939842697]",
            "(5916973408178703080,6050611292385734389]",
            "(6079154228273183729,6090089609473339695]",
            "(6233102408608229003,6238887659671392930]",
            "(6268397881596606388,6283167365273706631]",
            "(6382424778322053954,6405352309981402563]",
            "(6425709650188168169,6494205821215136348]",
            "(6599388239037707951,6724195608022104060]",
            "(6877147673659306574,6925189137023589324]",
            "(7025662255343686174,7055807262727319424]",
            "(7099440082474576491,7114406049240854360]",
            "(7181079371809651303,7199576419099781804]",
            "(7214675372183956393,7232733341982721609]",
            "(7292775917038263233,7309980407472880887]",
            "(7351650085248126195,7359030485367457448]",
            "(7380227913635029036,7414154199197577355]",
            "(7547556318670138169,7570094251113682739]",
            "(7649427894846954898,7651810029329144495]",
            "(7667541804338737129,7710964422865265962]",
            "(7928030646662068947,8089838211260662478]",
            "(8105963935427010978,8206628778459195202]",
            "(8440992966323396669,8452098743862133946]",
            "(8486663669237009036,8490080457235737117]",
            "(8584200698150181853,8659664415469601414]",
            "(8701561361806505978,8771341666294407497]",
            "(8890621759510705487,8970419178269923673]",
            "(9073493434922174946,9116475822629024863]",
            "(9122063044812619144,9125478626552811469]",
            "(9183410537987850115,9223372036854775807]"
          ],
          "allocation_id" : {
            "id" : "dummy_alloc_id"
          }
        }
      ]
    }
  }
}

@vroyer
Copy link
Collaborator

vroyer commented Apr 26, 2018 via email

@zllovesuki
Copy link
Author

@vroyer Although, it does frequently gives {"acknowledged": false} to some operations

@vroyer vroyer closed this as completed in 15f8c58 May 4, 2018
vroyer pushed a commit that referenced this issue May 16, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants