index aanmaken met mapping werkt niet op amazon

[wires]

@bertspaan NOOIT ERRORS OPETEN

fix:
https://gist.github.com/wires/6aa3615cc99f47693fd5#file-default-mapping-js-L35

lowercase analyser was unknown..

[bertspaan]

😢

[wires]

So even though code has been added to histograph/io create the indices and their mappings before indexing the document, somehow (I suspect a race condition?...) this doesn't, sometimes a document is indexed before the index is created (and thus get's automatic mapping).

I've solved this for now by creating the indices in histograph/core, before bulk indexing.

This problem needs a better solution

[wires]

Actually, we now have this index creation in two places (here and io), needs to be removed

[wires]

I think this duplicate index creation is what is causing errors on AWS now

GRAPHMALIZER => 108 docs, 672.628264ms
Failed creating index
{ [Error: Bad Request] status: '400', message: 'Bad Request' } 'Bad Request'
Created all indices []
ES => 56 indexed, took 66ms, errors: false

[hpfast]

I think this is affecting me too. Is solving this a question of removing it in one or the other place?

[wires]

no, it is a complicated race condition that is not easily fixed without changing how the API/IO/redis work. In the past we used the default-mapping file (described somewhere in the installation manual) that you place in the ES config directory. On AWS this is not possible so we changed the software to create the index. The io project manages the creation and deletion of datasets, which involves creating an elasticsearch index, but when PITs/relations are pushed to the dataset io places them on the redis queue and they end up in the Graphmalizer, which then indices them into ES and this implicitly creates an index. When the indices are already there you might see a problem in the console, but it shouldn't cause a problem.

[hpfast]

When the indices are already there you might see a problem in the
console, but it shouldn't cause a problem.

ok, yeah, I saw that Graphmalizer was still proceeding. Maybe a false alarm
for me.

On Mon, Nov 23, 2015 at 2:28 PM, Jelle Herold notifications@github.com
wrote:

no, it is a complicated race condition that is not easily fixed without
changing how the API/IO/redis work. In the past we used the default-mapping
file (described somewhere in the installation manual) that you place in the
ES config directory. On AWS this is not possible so we changed the software
to create the index. The io project manages the creation and deletion of
datasets, which involves creating an elasticsearch index, but when
PITs/relations are pushed to the dataset io places them on the redis
queue and they end up in the Graphmalizer, which then indices them into ES
and this implicitly creates an index. When the indices are already there
you might see a problem in the console, but it shouldn't cause a problem.

—
Reply to this email directly or view it on GitHub
#72 (comment).

[wires]

On 23 Nov 2015, at 14:47, Hans Fast notifications@github.com wrote:

When the indices are already there you might see a problem in the
console, but it shouldn't cause a problem.

ok, yeah, I saw that Graphmalizer was still proceeding. Maybe a false alarm
for me.

exactly, it’s probably a false alarm :^)

sorry for the confusion

[sbocconi]

Is this still an issue or is it resolved?

[bertspaan]

Still an issue.

[sbocconi]

We are running our own instance of ES, so the problem of the default mapping is not applicable anymore. As far as I can see there is no problem in creating the indexes