Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Creation of ES index fails if first mapping contains a map #249

Closed
filthz opened this issue Dec 19, 2018 · 30 comments
Closed

Creation of ES index fails if first mapping contains a map #249

filthz opened this issue Dec 19, 2018 · 30 comments

Comments

@filthz
Copy link

filthz commented Dec 19, 2018 •
edited

Version 5.5.0.24 Assume 2 Tables are given:

CREATE TABLE devices (
id uuid,
configuration map < text, text >,
PRIMARY KEY (id)
);

CREATE TABLE audit (
id uuid,
action text,
nano_since_epoch bigint,
username text,
PRIMARY KEY (id)
);
  • Insert some values into the devices table, where the map is NOT empty.
  • Indexing in this order will work:
{
  "settings": {
    "keyspace":"KEYSPACE_PLACEHOLDER",
    "number_of_replicas" : 1
  },
  "mappings": {
    "audit" : {
      "discover":"^(?!message).*"
    },
    "devices" : {
      "discover":"^(?!message).*"
    }
}

Indexing in this order will fail:

{
  "settings": {
    "keyspace":"KEYSPACE_PLACEHOLDER",
    "number_of_replicas" : 1
  },
  "mappings": {
    "devices" : {
      "discover":"^(?!message).*"
    },
    "audit" : {
      "discover":"^(?!message).*"
    }
}
@vroyer
Copy link
Collaborator

vroyer commented Jan 10, 2019

Would you have the mapping error related to this issue ?
In ES 6.x+, indices have only one type, so you will have 2 indices pointing to the same keyspace, and this might be a workaround for this issue.

@filthz
Copy link
Author

filthz commented Jan 11, 2019

Hi Vincent,

the error message was like this (it´s an other table name but the same issue). Currently I am still not sure, what it the cause of the problem. In my case adding timeouts between adding mappings helped. When all of the mappings were in 1 call this problem occured each time.

2018-12-21 07:36:18,754 ERROR [CompactionExecutor:536] ElasticSecondaryIndex.java:544 addField submapper not found for nested field [id] 2018-12-21 07:36:18,757 INFO [CompactionExecutor:536] ElasticSecondaryIndex.java:518 addField updating mapping={"sensors":{"properties":{"device_data":{"type":"nested","cql_struct":"map","cql_collection":"singleton","properties":{"comm_nr":{"type":"keyword"}}}}}} 2018-12-21 07:36:48,758 ERROR [CompactionExecutor:536] ElasticSecondaryIndex.java:527 addField error while updating mapping org.elasticsearch.ElasticsearchTimeoutException: blocking update timeout at org.elasticsearch.cluster.service.ClusterService$BlockingActionListener.waitForUpdate(ClusterService.java:2244) at org.elasticsearch.cluster.service.ClusterService.blockingMappingUpdate(ClusterService.java:2283) at org.elassandra.index.ElasticSecondaryIndex$Context.addField(ElasticSecondaryIndex.java:520) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.buildContext(ElasticSecondaryIndex.java:2083) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.buildContext(ElasticSecondaryIndex.java:1644) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.index(ElasticSecondaryIndex.java:2169) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.index(ElasticSecondaryIndex.java:2153) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.index(ElasticSecondaryIndex.java:1644) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.write(ElasticSecondaryIndex.java:2136) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.write(ElasticSecondaryIndex.java:1644) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer.update(ElasticSecondaryIndex.java:1685) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer.finish(ElasticSecondaryIndex.java:1866) at java.lang.Iterable.forEach(Iterable.java:75) at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:642) at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:139) at org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1637) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) at java.lang.Thread.run(Thread.java:748)

@vroyer
Copy link
Collaborator

vroyer commented Feb 13, 2019

Hi,

You reported in your last message an error when inserting document while your initial issue was related to discover the mapping.

In order to reproduce your issues, please reports error message and logs for the mapping issue (initial error), and information to reproduce the second error message (the mapping and the insert statement).
Thanks.

@filthz
Copy link
Author

filthz commented Feb 13, 2019 via email

@filthz
Copy link
Author

filthz commented Mar 27, 2019

Hi,

I have to reopen this, as I encountered exactly the same issue with ES6 (elassandra 6.2.3.12).
When adding ES index for this table:

CREATE TABLE devices (
    id uuid,
    comm_nr text,
    configuration map<text,text>,
    customer text,
    description text,
    groups list<text>,
    last_active timeuuid,
    last_active_nanoseconds bigint,
    location uuid,
    notes text,
    picture text,
    status text,
    tag text,
    thumbnail text,
    "type" text,
    "users" list<text>,
    PRIMARY KEY (id)
)

It fails. In system.log I see:

2019-03-26 21:58:43,974 INFO [elasticsearch[10.164.0.22][generic][T#4]] CassandraShardStateListener.java:59 afterIndexShardStarted shard [devices][0] started 2019-03-26 21:58:43,975 INFO [elasticsearch[10.164.0.22][generic][T#4]] ElasticSecondaryIndex.java:2545 startRebuildIfNeeded start building secondary iot_data.devices.elastic_devices_idx 2019-03-26 21:58:44,002 INFO [elasticsearch[10.164.0.22][generic][T#4]] SecondaryIndexManager.java:571 buildIndexesBlocking Submitting index build of elastic_devices_idx for data in BigTableReader(path='/var/lib/cassandra/data/iot_data/devices-c4a249f04ff611e9ba438dba5fe3507f/mc-1-big-Data.db') 2019-03-26 21:58:44,010 INFO [CompactionExecutor:74] ElasticSecondaryIndex.java:490 addField updating mapping={"devices":{"properties":{"configuration":{"type":"nested","cql_struct":"map","cql_collection":"singleton","properties":{"size":{"type":"keyword"}}}}}} 2019-03-26 21:58:44,046 INFO [elasticsearch[10.164.0.22][masterService#updateTask][T#1]] CassandraDiscovery.java:920 publish CQL SchemaChanges=[UPDATED TABLE iot_data.devices, UPDATED TABLE elastic_admin.metadata] 2019-03-26 21:59:14,041 INFO [elasticsearch[10.164.0.22][masterService#updateTask][T#1]] CassandraDiscovery.java:964 publish commit state [6d18bf5c-32df-4815-bf06-e565cc71ac9b/7] timeout with pending nodes=[{10.164.0.22}{6d18bf5c-32df-4815-bf06-e565cc71ac9b}{6d18bf5c-32df-4815-bf06-e565cc71ac9b}{10.164.0.22}{10.164.0.22:9300}{ALIVE}{rack=rack1, dc=datacenter-prod-1}] 2019-03-26 21:59:14,042 WARN [elasticsearch[10.164.0.22][masterService#updateTask][T#1]] MasterService.java:583 warnAboutSlowTaskIfNeeded cluster state update task [put-mapping[devices]] took [30s] above the warn threshold of 30s 2019-03-26 21:59:14,042 ERROR [CompactionExecutor:74] ElasticSecondaryIndex.java:499 addField error while updating mapping org.elasticsearch.ElasticsearchTimeoutException: blocking update timeout at org.elasticsearch.cluster.service.ClusterService$BlockingActionListener.waitForUpdate(ClusterService.java:640) at org.elasticsearch.cluster.service.ClusterService.blockingMappingUpdate(ClusterService.java:689) at org.elassandra.index.ElasticSecondaryIndex.addField(ElasticSecondaryIndex.java:492) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.buildContext(ElasticSecondaryIndex.java:2192) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.buildContext(ElasticSecondaryIndex.java:1738) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.index(ElasticSecondaryIndex.java:2285) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.index(ElasticSecondaryIndex.java:2262) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.index(ElasticSecondaryIndex.java:1738) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.write(ElasticSecondaryIndex.java:2245) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.write(ElasticSecondaryIndex.java:1738) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer.update(ElasticSecondaryIndex.java:1779) at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer.finish(ElasticSecondaryIndex.java:1962) at java.lang.Iterable.forEach(Iterable.java:75) at org.apache.cassandra.index.SecondaryIndexManager.indexPartition(SecondaryIndexManager.java:1067) at org.apache.cassandra.index.internal.CollatedViewIndexBuilder.build(CollatedViewIndexBuilder.java:139) at org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1637) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) at java.lang.Thread.run(Thread.java:748) 2019-03-26 21:59:14,043 ERROR [CompactionExecutor:74] ElasticSecondaryIndex.java:516 addField submapper not found for nested field [size] in index [devices] 2019-03-26 21:59:14,044 INFO [CompactionExecutor:74] ElasticSecondaryIndex.java:490 addField updating mapping={"devices":{"properties":{"configuration":{"type":"nested","cql_struct":"map","cql_collection":"singleton","properties":{"pressure":{"type":"keyword"}}}}}} 2019-03-26 21:59:14,086 INFO [elasticsearch[10.164.0.22][masterService#updateTask][T#1]] CassandraDiscovery.java:920 publish CQL SchemaChanges=[UPDATED TABLE iot_data.devices, UPDATED TABLE elastic_admin.metadata] 2019-03-26 21:59:44,045 ERROR [CompactionExecutor:74] ElasticSecondaryIndex.java:499 addField error while updating mapping org.elasticsearch.ElasticsearchTimeoutException: blocking update timeout

This error continues then. From the logs I see that it complains about the "configuration" field of my device entity. Here is an example of an real entry for this field:

{'datasource' : 'api', 'external_id' : '003011FADB4A', 'first_active_on' : '01-03-2019', 'pressure' : '12S', 'size' : '35'}

When I add the mapping with "configuration" column ignored, there are no errors.

Please help

@vroyer
Copy link
Collaborator

vroyer commented Apr 1, 2019

Hi,

Yes, there is an issue when creating the index with a populated map.
Currently, the workaround is to create your index without indexing the map (the configuration field in your case) to initialize the index. Then update the mapping to index the map field, and if needed, rebuild the index to get the map field content indexed. Alternatively, you create your index with an explicit mapping of all map keys (datasource, external_id, first_active_on, pressure, size in your case).

To create your index without mapping the field configuration:
curl -XPUT -H 'Content-Type: application/json' https://localhost:9200/toto -d'{"mappings":{"devices":{"discover":"^((?!configuration).*)"}}}'

Index should be green at this step, then to update mapping to add the configuration field:
curl -XPUT -H 'Content-Type: application/json' https://localhost:9200/toto/_mapping/devices -d'{"discover":".*"}'

Then on every nodes, run the following command to rebuild the index:
nodetool rebuild_index devices elastic_devices_idx

Thank's for reporting this issue.

@filthz
Copy link
Author

filthz commented May 17, 2019

Hi Vincent,

could you tell me, if this has been resolved in current releases?

Thanks!
Alex

@vroyer
Copy link
Collaborator

vroyer commented May 17, 2019

Hi Alex,
According to my tests, it good in version 6.2.3.15
Let me know if it works in your use case.
Thanks.

@filthz
Copy link
Author

filthz commented May 27, 2019

Hi Vincent,

I am testing this with 6.2.3.15, the reported issue seems to be resolved.
Anyways I am getting an other error when I try to insert an entry:

{
  "it-alarms" : {
    "aliases" : { },
    "mappings" : {
      "alarms" : {
        "properties" : {
          "acknowledged" : {
            "type" : "boolean",
            "cql_collection" : "singleton"
          },
          "description" : {
            "type" : "keyword",
            "cql_collection" : "singleton"
          },
          "device_comm_nr" : {
            "type" : "keyword",
            "cql_collection" : "singleton"
          },
          "device_id" : {
            "type" : "keyword",
            "cql_collection" : "singleton"
          },
          "groups" : {
            "type" : "keyword"
          },
          "id" : {
            "type" : "keyword",
            "cql_collection" : "singleton",
            "cql_partition_key" : true,
            "cql_primary_key_order" : 0
          },
          "level" : {
            "type" : "keyword",
            "cql_collection" : "singleton"
          },
          "nano_since_epoch" : {
            "type" : "long",
            "cql_collection" : "singleton"
          },
          "source_description" : {
            "type" : "keyword",
            "cql_collection" : "singleton"
          },
          "source_id" : {
            "type" : "keyword",
            "cql_collection" : "singleton"
          },
          "source_type" : {
            "type" : "keyword",
            "cql_collection" : "singleton"
          },
          "threshold_max" : {
            "type" : "double",
            "cql_collection" : "singleton"
          },
          "threshold_min" : {
            "type" : "double",
            "cql_collection" : "singleton"
          },
          "users" : {
            "type" : "keyword"
          },
          "value" : {
            "type" : "double",
            "cql_collection" : "singleton"
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "keyspace" : "testkeyspace",
        "number_of_shards" : "1",
        "provided_name" : "it-alarms",
        "creation_date" : "1558965933641",
        "number_of_replicas" : "0",
        "uuid" : "JOSnbbnSTruYSlWWrH6nkQ",
        "version" : {
          "created" : "6020399"
        }
      }
    }
  }
}

The error I get:

2019-05-27 16:05:43,635 ERROR [MutationStage-4] ElasticSecondaryIndex.java:2246 write Unexpected error
java.lang.NullPointerException: null
at org.elasticsearch.index.mapper.UidFieldMapper.createField(UidFieldMapper.java:236)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.buildContext(ElasticSecondaryIndex.java:2169)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.buildContext(ElasticSecondaryIndex.java:1730)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.index(ElasticSecondaryIndex.java:2287)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.index(ElasticSecondaryIndex.java:2258)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.index(ElasticSecondaryIndex.java:1730)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer$Rowcument.write(ElasticSecondaryIndex.java:2241)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer$SkinnyRowcument.write(ElasticSecondaryIndex.java:1730)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$SkinnyRowcumentIndexer.update(ElasticSecondaryIndex.java:1771)
at org.elassandra.index.ElasticSecondaryIndex$ImmutableMappingInfo$RowcumentIndexer.finish(ElasticSecondaryIndex.java:1955)
at org.apache.cassandra.index.SecondaryIndexManager$WriteTimeTransaction.commit(SecondaryIndexManager.java:1432)
at org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:181)
at org.apache.cassandra.db.Memtable.put(Memtable.java:282)
at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1359)
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:631)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:475)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:232)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:241)
at org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1398)
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2637)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:114)
at java.lang.Thread.run(Thread.java:748)

I am inserting an empty entry, which contains only the id.

@filthz
Copy link
Author

filthz commented May 29, 2019 •
edited

Hi,

after some more testing I can confirm, that the bug is still not fixed.
Prepopulated maps are not being indexed correctly and break.

This issue does not occur on a 1 node cluster, i was only able to observe it on a 2 node cluster with RF=2.
That even occurs with your proposed work around.

Please please fix this

Best regards

@vroyer
Copy link
Collaborator

vroyer commented Jun 1, 2019

The original bug is fixed, you get here another issue because your index was probably created with version 5.x (with multiple types per index). As elasticsearch 6.x, Elassandra only supports single type index, and Uid is only indexed in multiple types index (index < 6.0).

The next version if fixed to avoid NPE, but we do not support hot ES index upgrade, because you can smoothly upgrade with multiple Cassandra datacenters...

@filthz
Copy link
Author

filthz commented Jun 1, 2019 via email

@vroyer
Copy link
Collaborator

vroyer commented Jun 25, 2019

Deleting the elastic directory does not delete your index settings where index version is recorded.
I don't really know how your recreate your index, but the error (NPE in UidFieldMapper.java:236) cannot happen with ES 6.x index. We need more details about this issue to explain or solve it.
Thanks.

@filthz
Copy link
Author

filthz commented Jun 26, 2019 via email

@vroyer
Copy link
Collaborator

vroyer commented Jul 2, 2019

1.Delete all you ES indices
2.Restart all nodes with Elasticsearch disbaled (mainclass=CassandraDaemon in /etc/default/cassandra)
3.Drop the elastic_admin keyspace
4. Check /var/lib/cassandra/elasticsearch.data is empty
5.Upgrade the elassandra binary from 5.x to 6.2.3.17 on all nodes
6.Restart all nodes with Elasticsearch enabled (mainclass=ElassandraDaemon in /etc/default/cassandra)
7.Recreate your ES indices with a new mapping for ES 6.x

@filthz
Copy link
Author

filthz commented Aug 13, 2019

For me this issue is still there with 6.2.3.18
Steps to reproduce:

  1. Fresh install without any data.
  2. Import data containing Map<String,String>
  3. Start 3 nodes in total
  4. change replication factor to 3
  5. nodetool repair on node2 + node3
  6. observe mapping errors in the logs

Note that this only occured as 3 nodes were started.
We have stopped using maps inside entities --> errors gone

@vroyer
Copy link
Collaborator

vroyer commented Aug 13, 2019

Could you attach you mapping, your CQL schema and the logs containing errors. Could also provide the exact nodetool repair command you ran after changing the RF from 1 (i guess) to 3. Thanks.

@filthz
Copy link
Author

filthz commented Aug 13, 2019

Error:
2019-08-13 07:09:35,988 WARN [elasticsearch[10.164.0.58][masterService#updateTask][T#1]] MasterService.java:282 runTasks failed to publish updated cluster state in [5s]: version [27], uuid [ASEADRK9SS-mPIOA6JLizg], source [put-mapping[devices]] cluster uuid: ae912935-612b-4ae6-ae2e-0c81126923cc version: 27 state uuid: ASEADRK9SS-mPIOA6JLizg from_diff: false meta data version: 75 [audit/XK9AXfrkRlmeGSQmSfRw7Q]: v[3] 0: p_term [0], isa_ids [] [it-derived_value_configuration_by_device/X6GGxLK2Rp60H43AlRi-Hw]: v[3] 0: p_term [0], isa_ids [] [alarms/jS5RrPqyRpKGJwKrWY5LJg]: v[3] 0: p_term [0], isa_ids [] [alarm_configurations/9nj0t0XXQCyrPeWloP1Drw]: v[3] 0: p_term [0], isa_ids [] [it-alarm_configurations/jGOV_Z4ISpaRqXMFj4u7WA]: v[3] 0: p_term [0], isa_ids [] [users/QFkU0VTmR_S2iWkWj6Jw4w]: v[3] 0: p_term [0], isa_ids [] [derived_value_configuration_by_device/R6hzV_DDSSWTxq_8XFi3bg]: v[4] 0: p_term [0], isa_ids [] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses. at org.apache.cassandra.service.paxos.AbstractPaxosCallback.await(AbstractPaxosCallback.java:64) at org.apache.cassandra.service.StorageProxy.preparePaxos(StorageProxy.java:489) at org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:402) at org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:237) at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:449) at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:424) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:225) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:256) at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:241) at org.elasticsearch.cluster.service.ClusterService.process(ClusterService.java:536) at org.elasticsearch.cluster.service.ClusterService.processWriteConditional(ClusterService.java:548) at org.elasticsearch.cluster.service.ClusterService.commitMetaData(ClusterService.java:1264) at org.elassandra.discovery.CassandraDiscovery.publishAsCoordinator(CassandraDiscovery.java:1002) ... 11 common frames omitted 2019-08-13 07:09:25,919 ERROR [MutationStage-32] ElasticSecondaryIndex.java:508 addField submapper not found for nested field [failedPacketsSinceStart] in index [devices]

(this continues with other tables containing maps, which are indexed in elastic as well).
failedPacketsSinceStart is a key inside the map in the devices table.

Devices Schema:
CREATE TABLE devices ( id uuid, children list<uuid>, comm_nr text, configuration map<text,text>, create_date_nanoseconds bigint, customer text, description text, groups list<text>, last_active text, last_active_nanoseconds bigint, location uuid, notes text, parent uuid, picture text, status text, tag text, thumbnail text, "type" text, "users" list<text>, PRIMARY KEY (id) )

Repair command:
nodetool repair

@vroyer
Copy link
Collaborator

vroyer commented Aug 13, 2019 via email

@filthz
Copy link
Author

filthz commented Aug 13, 2019

Hi Vincent,

i hope I dont mix things up. What happened is that our 5.5.0.24 cluster crashed with almost the same error but without an repair. In this state its impossible to delete the elastic index so you are kinda doomed.

I started a 1 node cluster, imported 2 tables - both of them are small (<100) entries.
Then I started a second node. Changed RF=2. nodetool repair on node 2.
Then 3rd node with the same procedure and the errors came. It might be a coincidence but these errors are there.

The data was imported via the COPY command from .csv files, if that is important. I can even provide you the .csv data if it helps.

Thanks for looking into that
Alex

@vroyer
Copy link
Collaborator

vroyer commented Aug 14, 2019 via email •
edited

@filthz
Copy link
Author

filthz commented Aug 14, 2019 •
edited

Hi Vincent,

i first imported the data (the map field is filled). Then i created the index with:

{ "settings" : { "keyspace" : "iot_data" }, "mappings" : { "devices" : { "discover":"^(?!message).*" } } }
cas_contention_timeout_in_ms - is the default value not 1000 already?

Alex

@filthz
Copy link
Author

filthz commented Oct 4, 2019

Any news here?

@filthz
Copy link
Author

filthz commented Oct 12, 2019

bump

@vroyer
Copy link
Collaborator

vroyer commented Oct 13, 2019 •
edited

The default cas_contention_timeout_in_ms is 5000, but you should check your conf/cassandra.yaml file.

Release 6.2.3.19 adds a new way to index map, with opaque_map, there is no more Elasticsearch mapping update (and no more PAXOS timeout). If you don't need to discover indexed fields through the elasticsearch mapping, this is probably a nice solution in your case. See documentation for details.

@filthz
Copy link
Author

filthz commented Oct 13, 2019

Thank you!
Do i understand it right, that opaque_map is fully searchable?

So searching with:

curl -XGET  -H 'Content-Type: application/json' "http://localhost:9200/twitter/_search?pretty=true" -d '{
"query":{
    "nested":{
            "path":"attrs",
            "query":{ "term": {"attrs.city":"paris" } }
             }
        }
}

will be possible?

@vroyer
Copy link
Collaborator

vroyer commented Oct 13, 2019

Yes is fully searchable, you just need to know what are your field names (or key entries in the underlying cassandra map), as it is not more visible in the elasticsearch mapping.

In the next release 6.2.3.20 (next week), aggregation on opaque_map keys will be supported too.

@filthz
Copy link
Author

filthz commented Oct 13, 2019

Sounds awesome!

@filthz
Copy link
Author

filthz commented Oct 23, 2019

Is there a specific update procedure to 6.2.3.20 when no maps are inside of any index?

@filthz
Copy link
Author

filthz commented Apr 17, 2020

I can confirm this is fixed

@vroyer vroyer closed this as completed Jun 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@filthz @vroyer