Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEATURE: Adapt indexing to create one index per dimensions preset #298

Open
wants to merge 31 commits into
base: master
from

Conversation

@dfeyer
Copy link
Member

commented Apr 5, 2019

This change also use sub command to lower to amount of memory needed by a full indexing.

Features

  • Sub command to limit memory usage
  • Split bulk request based on size/length limits
  • One index per node combination hash

Todo

  • Split indexing across multiple indexes
  • Switch index, based on context node target dimensions, during query
  • Use sub command during nodeindex:build to avoid high memory usage
  • Index invisible content
  • CLI command to show mapping between index name and dimensions preset
  • Add support for per index configuration (analyzer, stemming, ...)
  • Use the old configuration for all indices if no more specific configuration provided (never merge the old and the new config)
  • Update documentation
  • Fix tests and improve test coverage
TASK: Adapt indexing to create one indexes per dimensions preset
This change also use sub command to lower to amount of memory needed by a full indexing.

@dfeyer dfeyer self-assigned this Apr 5, 2019

@dfeyer

This comment has been minimized.

Copy link
Member Author

commented Apr 5, 2019

This is the first part of the change, should contains bugs ;) but at least you should see multiple indexes created after a full indexing, like this for the Demo site:

yellow open typo3cr-fb11fdde869d0a8fcfe00a2fd35c031d-1554475965 YaeSswPpSPqBBUrMuOQVDQ 5 1  14  14  87.3kb  87.3kb
yellow open typo3cr-0359ed5c416567b8bc2e5ade0f277b36-1554475965 VHaCW0FjSh-TDR7E5XswhQ 5 1 568 121     3mb     3mb
yellow open typo3cr-8f21da181f6ce38197fcb78d49f5384e-1554475965 jIATRNRxR7Gb0Gb8hHHVNw 5 1  14  14  87.8kb  87.8kb
yellow open typo3cr-00f3cd3ea8f2654d5c7e4c27146b525a-1554475965 CMhWGqoNSzS8htRX7TUwLQ 5 1  14  18  94.8kb  94.8kb
yellow open typo3cr-e28daf5b04394a7b506534acc0ad491a-1554475965 zV93sNTjRrm0Fwz9QBEWPg 5 1  14  16  75.8kb  75.8kb
yellow open typo3cr-4f534b1eb0c1a785da31e681fb5e91ff-1554475965 SXesDiOVRFej8yaiXYFBiw 5 1 494 750   4.4mb   4.4mb
yellow open typo3cr-74fef9b79f12e578c9e7af76f2ed7aaa-1554475965 5HDuiwKSTVaSr1gj-VLV-w 5 1  78  44 307.8kb 307.8kb
dfeyer added 4 commits Apr 5, 2019
TASK: Remove __dimensionCombinationHash
This field is not used anymore. The index contains only one dimension combination
@dfeyer

This comment has been minimized.

Copy link
Member Author

commented Apr 5, 2019

Tests are broken, but this could works, I need to update the doc with the change

Index alias changes

The typo3cr (by the way we can use the next major to get ride of the name), is an index that point to all dimensions preset specific index, not sure if it's a good idea, ... or even useful. The aliases looks like this now:

typo3cr                                  typo3cr-8f21da181f6ce38197fcb78d49f5384e-1554494013 - - -
typo3cr-1554494013                       typo3cr-8f21da181f6ce38197fcb78d49f5384e-1554494013 - - -
typo3cr-8f21da181f6ce38197fcb78d49f5384e typo3cr-8f21da181f6ce38197fcb78d49f5384e-1554494013 - - -
typo3cr                                  typo3cr-fb11fdde869d0a8fcfe00a2fd35c031d-1554494013 - - -
typo3cr-1554494013                       typo3cr-fb11fdde869d0a8fcfe00a2fd35c031d-1554494013 - - -
typo3cr-fb11fdde869d0a8fcfe00a2fd35c031d typo3cr-fb11fdde869d0a8fcfe00a2fd35c031d-1554494013 - - -
typo3cr                                  typo3cr-e28daf5b04394a7b506534acc0ad491a-1554494013 - - -
typo3cr-1554494013                       typo3cr-e28daf5b04394a7b506534acc0ad491a-1554494013 - - -
typo3cr-e28daf5b04394a7b506534acc0ad491a typo3cr-e28daf5b04394a7b506534acc0ad491a-1554494013 - - -
typo3cr                                  typo3cr-4f534b1eb0c1a785da31e681fb5e91ff-1554494013 - - -
typo3cr-1554494013                       typo3cr-4f534b1eb0c1a785da31e681fb5e91ff-1554494013 - - -
typo3cr-4f534b1eb0c1a785da31e681fb5e91ff typo3cr-4f534b1eb0c1a785da31e681fb5e91ff-1554494013 - - -
typo3cr                                  typo3cr-00f3cd3ea8f2654d5c7e4c27146b525a-1554494013 - - -
typo3cr-00f3cd3ea8f2654d5c7e4c27146b525a typo3cr-00f3cd3ea8f2654d5c7e4c27146b525a-1554494013 - - -
typo3cr-1554494013                       typo3cr-00f3cd3ea8f2654d5c7e4c27146b525a-1554494013 - - -
typo3cr                                  typo3cr-0359ed5c416567b8bc2e5ade0f277b36-1554494013 - - -
typo3cr-0359ed5c416567b8bc2e5ade0f277b36 typo3cr-0359ed5c416567b8bc2e5ade0f277b36-1554494013 - - -
typo3cr-1554494013                       typo3cr-0359ed5c416567b8bc2e5ade0f277b36-1554494013 - - -
typo3cr                                  typo3cr-74fef9b79f12e578c9e7af76f2ed7aaa-1554494013 - - -
typo3cr-1554494013                       typo3cr-74fef9b79f12e578c9e7af76f2ed7aaa-1554494013 - - -
typo3cr-74fef9b79f12e578c9e7af76f2ed7aaa typo3cr-74fef9b79f12e578c9e7af76f2ed7aaa-1554494013 - - -

Indices naming

The index name use a hash, unique per dimensions preset. You can use the CLI command to have a better understanding of the relation between the hash and the dimension presets:

~/P/s/g/d/neos-flowpack-elasticsearch » ./flow nodeindex:showdimensionsmapping              master ✗
typo3cr-4f534b1eb0c1a785da31e681fb5e91ff {"language":["en_US"]}
typo3cr-0359ed5c416567b8bc2e5ade0f277b36 {"language":["en_UK","en_US"]}
typo3cr-fb11fdde869d0a8fcfe00a2fd35c031d {"language":["de"]}
typo3cr-8f21da181f6ce38197fcb78d49f5384e {"language":["fr"]}
typo3cr-e28daf5b04394a7b506534acc0ad491a {"language":["nl","de"]}
typo3cr-00f3cd3ea8f2654d5c7e4c27146b525a {"language":["da"]}
typo3cr-74fef9b79f12e578c9e7af76f2ed7aaa {"language":["lv"]}

Affected Interface

Only one affected Interface: Classes/Driver/IndexDriverInterface.php

Configuration change

We can now use the specific index name to have by example language stemming or synonyms per language.

I need to check if we can use the old configuration, based on the index name with the dimensions preset suffix for all more specific index, by default, that can make the change less breaking.

@dfeyer

This comment has been minimized.

Copy link
Member Author

commented Apr 5, 2019

If some of you have time to test with different dimensions configuration (and also without dimensions, currently not tested at all)

@dfeyer dfeyer changed the title TASK: Adapt indexing to create one indexes per dimensions preset TASK: Adapt indexing to create one index per dimensions preset Apr 5, 2019

Classes/Driver/IndexDriverInterface.php Outdated Show resolved Hide resolved
@@ -81,9 +81,6 @@ public function buildMappingInformation(Index $index): MappingCollection
'type' => 'text'
]
]);
$mapping->setPropertyByPath('__dimensionCombinationHash', [

This comment has been minimized.

Copy link
@dfeyer

dfeyer Apr 5, 2019

Author Member

This is not needed any more

if (is_array($dimensionCombinations)) {
$this->queryFilter('term', ['__dimensionCombinationHash' => md5(json_encode($dimensionCombinations))]);
}

This comment has been minimized.

Copy link
@dfeyer

dfeyer Apr 5, 2019

Author Member

Not needed anymore

@@ -758,6 +758,8 @@ public function moreLikeThis(array $like, array $fields = [], array $options = [
*/
public function query(NodeInterface $contextNode)
{
$this->elasticSearchClient->setDimensions($contextNode->getContext()->getTargetDimensions());

This comment has been minimized.

Copy link
@dfeyer

dfeyer Apr 5, 2019

Author Member

This parts is the weekiest code of the PR ... you MUST do that before querying to switch the the correct index. I will try to design a ÌndexSelectorStrategyInterface in the Flowpack.ElasticSearch package, to have a central place where you can configure how the switch is done. But I think a bit of sleep before ;)

This comment has been minimized.

Copy link
@dfeyer

dfeyer Apr 5, 2019

Author Member

I think I have a solution ElasticSearchClient MUST store the context node (actually the context node is stored in the QueryBuilder, like this when calling $this->elasticSearchClient->getIndex()->request() we can switch the index. In the case the context node is required, and we need to test feature, like count, more like this....

This comment has been minimized.

Copy link
@dfeyer

dfeyer Apr 6, 2019

Author Member

Done in 80a2e15

@@ -243,7 +249,9 @@ public function indexNode(NodeInterface $node, $targetWorkspaceName = null)
$handleNode = function (NodeInterface $node, Context $context) use ($targetWorkspaceName, $indexer) {
$nodeFromContext = $context->getNodeByIdentifier($node->getIdentifier());
if ($nodeFromContext instanceof NodeInterface) {
$indexer($nodeFromContext, $targetWorkspaceName);
$this->searchClient->withDimensions(function () use ($indexer, $nodeFromContext, $targetWorkspaceName) {

This comment has been minimized.

Copy link
@dfeyer

dfeyer Apr 5, 2019

Author Member

This where we switch the index during the indexing

/**
* Update the main alias to allow to query all indices at once
*/
public function updateMainAlias()

This comment has been minimized.

Copy link
@dfeyer

dfeyer Apr 5, 2019

Author Member

Not sure if this is a good idea or not

dfeyer added 7 commits Apr 6, 2019
TASK: Extract CLI related to mapping (index name and node type mapping)
This change create a new CLI controller for commands related to the mapping of the
index names and the node type configuration.
Classes/Driver/IndexDriverInterface.php Outdated Show resolved Hide resolved
Classes/Indexer/WorkspaceIndexer.php Show resolved Hide resolved
Classes/Indexer/WorkspaceIndexer.php Outdated Show resolved Hide resolved
Classes/Indexer/WorkspaceIndexer.php Outdated Show resolved Hide resolved
Classes/Indexer/WorkspaceIndexer.php Outdated Show resolved Hide resolved
Classes/Command/NodeIndexCommandController.php Outdated Show resolved Hide resolved
Classes/Command/NodeIndexCommandController.php Outdated Show resolved Hide resolved
Classes/Command/NodeIndexCommandController.php Outdated Show resolved Hide resolved
Classes/Command/NodeIndexCommandController.php Outdated Show resolved Hide resolved
Classes/Command/NodeIndexCommandController.php Outdated Show resolved Hide resolved
@daniellienert

This comment has been minimized.

Copy link
Contributor

commented Apr 7, 2019

Hey Dominique, tested the indexing with two dimension. The indexes are created successfully so far.
But it seems that index configuration is not applied correctly.

I've tested with an alternative index name and:

    indexes:
      default:
        typo3cr: ~
        punktde:
          index:
            number_of_shards: 1
            number_of_replicas: 0

but the indexes are configured with 5 shards / replicas. That works on master

dfeyer and others added 4 commits Apr 7, 2019
TASK: Add declare static in WorkspaceIndexer
Co-Authored-By: dfeyer <dfeyer@ttree.ch>
TASK: Add type casting in WorkspaceIndexer::indexWithDimensions
Co-Authored-By: dfeyer <dfeyer@ttree.ch>
@dfeyer

This comment has been minimized.

Copy link
Member Author

commented Apr 8, 2019

Hey Dominique, tested the indexing with two dimension. The indexes are created successfully so far.
But it seems that index configuration is not applied correctly.

Thanks for the review, custom index configuration are not tested currently and need a bit more love

@daniellienert

This comment has been minimized.

Copy link
Contributor

commented May 17, 2019

Hey @dfeyer - do you have any plans to progress on this? I am currently thinking about making the package compatible to ES 6+ which could cause lots if merge conflicts. Merging this feature at first would avoid this.

@DrillSergeant

This comment has been minimized.

Copy link

commented May 21, 2019

Would be so good to have this feature ready. Here in Switzerland we really need this a our customers are often multilingual and need to have well configured stemming/analysis configured per language.

I would love to help here by testing. Currently I struggle to get a ES5-docker container running, I have a running ES6-container with no problems, so a version that would run with ES6 would be great.

@dfeyer dfeyer changed the title TASK: Adapt indexing to create one index per dimensions preset FEATURE: Adapt indexing to create one index per dimensions preset May 22, 2019

@dfeyer

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

But it seems that index configuration is not applied correctly.

@daniellienert Actually it MUST work, but you MUST use the full index name:

Flowpack:
  ElasticSearch:

    transfer:
      connectionTimeout: 3600

    indexes:
      default:
        # language: fr
        'techspirit-8f21da181f6ce38197fcb78d49f5384e':
          analysis:
            filter:
              ngram_base:
                type: nGram
                min_gram: 2
                max_gram: 20
                token_chars: ['letter', 'digit', 'punctuation', 'symbol']
              elision:
                type: 'elision'
                articles: [ 'l', 'm', 't', 'qu', 'n', 's', 'j', 'd' ]

I need to implement the fallback on the short index name, only if the full index name does not have a specific configuration. That's in the todos.

@daniellienert Can you confirm that it work for your, with the full index name ?

@dfeyer

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

I would love to help here by testing. Currently I struggle to get a ES5-docker container running, I have a running ES6-container with no problems, so a version that would run with ES6 would be great.

@DrillSergeant This PR will not include ES6 support, it's planned for an other change, sorry ... maybe I can help you to setup a ES5 in docker ? What your problem currently ?

@dfeyer

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

Hey @dfeyer - do you have any plans to progress on this? I am currently thinking about making the package compatible to ES 6+ which could cause lots if merge conflicts. Merging this feature at first would avoid this.

Currently working on it, hope to finish the last missing feature and a lots of testing during next week

@DrillSergeant

This comment has been minimized.

Copy link

commented May 23, 2019

I would love to help here by testing. Currently I struggle to get a ES5-docker container running, I have a running ES6-container with no problems, so a version that would run with ES6 would be great.

@DrillSergeant This PR will not include ES6 support, it's planned for an other change, sorry ... maybe I can help you to setup a ES5 in docker ? What your problem currently ?

I got the problem solved yesterday, ES5 container running in ddev now. Don't know what was the problem, after removeing the es-volume and restarting the container, it worked.

@DrillSergeant

This comment has been minimized.

Copy link

commented Jun 6, 2019

Error with field __completion

when I try to run nodeindex:build I run into:

uid_1000@c61c4aad4218:/var/www/html$ ./flow nodeindex:build

Create indicies ...

Created index siganeos4-bf08e5d95164c1e4f496af81438ca9ca-1559741109
+ Dimensions: {"country":["global"],"language":["en"]}
Elasticsearch request failed.
[PUT
http://elastic:changeme@siganeos4.ddev.local:9200/siganeos4-bf08e5d95164c1e4f496af81438ca9ca-1559741109/Internezzo-Neos:DealerLocator/_mapping]:
Array
(
    [root_cause] => Array
        (
            [0] => Array
                (
                    [type] => mapper_parsing_exception
                    [reason] => Mapping definition for [__completion] has
unsupported parameters:  [analyzer : autocomplete]
                )

        )

    [type] => mapper_parsing_exception
    [reason] => Mapping definition for [__completion] has unsupported
parameters:  [analyzer : autocomplete]
)
; Response body:
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Mapping
definition for [__completion] has unsupported parameters:  [analyzer :
autocomplete]"}],"type":"mapper_parsing_exception","reason":"Mapping
definition for [__completion] has unsupported parameters:  [analyzer :
autocomplete]"},"status":400}

Request data:
{"Internezzo-Neos:DealerLocator":{"dynamic_templates":[],"properties":{"_creationDateTime":{"type":"date","format":"date_time_no_millis"},"_lastModificationDateTime":{"type":"date","format":"date_time_no_millis"},"_lastPublicationDateTime":{"type":"date","format":"date_time_no_millis"},"_path":{"type":"keyword","index":true},"_name":{"type":"keyword","index":true},"_nodeType":{"type":"keyword","index":true},"__identifier":{"type":"keyword","index":true},"__workspace":{"type":"keyword","index":true},"__path":{"type":"keyword","index":true},"__parentPath":{"type":"keyword","index":true},"__sortIndex":{"type":"integer"},"__typeAndSupertypes":{"type":"keyword","index":true},"_hidden":{"type":"boolean"},"_hiddenBeforeDateTime":{"type":"date","format":"date_time_no_millis"},"_hiddenAfterDateTime":{"type":"date","format":"date_time_no_millis"},"__completion":{"type":"keyword","analyzer":"autocomplete","index":true},"titleOverride":{"type":"keyword","index":true},"metaDescription":{"type":"keyword","index":true},"metaKeywords":{"type":"keyword","index":true},"metaRobotsNoindex":{"type":"boolean"},"metaRobotsNofollow":{"type":"boolean"},"twitterCardType":{"type":"keyword","index":true},"twitterCardCreator":{"type":"keyword","index":true},"twitterCardTitle":{"type":"keyword","index":true},"twitterCardDescription":{"type":"keyword","index":true},"canonicalLink":{"type":"keyword","index":true},"openGraphType":{"type":"keyword","index":true},"openGraphTitle":{"type":"keyword","index":true},"openGraphDescription":{"type":"keyword","index":true},"xmlSitemapChangeFrequency":{"type":"keyword","index":true},"xmlSitemapPriority":{"type":"keyword","index":true},"title":{"type":"keyword","index":true},"uriPathSegment":{"type":"keyword","index":true},"_hiddenInIndex":{"type":"boolean"},"__fulltextParts":{"type":"object","enabled":false},"__fulltext":{"type":"object","properties":{"h1":{"type":"text","index":true,"boost":20},"h2":{"type":"text","index":true,"boost":12},"h3":{"type":"text","index":true,"boost":10},"h4":{"type":"text","index":true,"boost":5},"h5":{"type":"text","index":true,"boost":3},"h6":{"type":"text","index":true,"boost":2},"text":{"type":"text","index":true,"boost":1,"analyzer":"language"}}},"hiddenInSearch":{"type":"boolean"},"lead":{"type":"keyword","index":true},"teaserTitle":{"type":"keyword","index":true},"teaserText":{"type":"keyword","index":true},"headline":{"type":"keyword","index":true},"headerImageAlt":{"type":"keyword","index":true},"headerImageTitle":{"type":"keyword","index":true},"headerImageCaption":{"type":"keyword","index":true}}}}

As I dont need autocompletion (yet) I could get rid of this problem by disabling the mixin, but the issue may remain if you need autocompletion.
My workaround:

'Neos.Neos:Document':
  superTypes:
    'Flowpack.SearchPlugin:AutocompletableMixin': false
@DrillSergeant

This comment has been minimized.

Copy link

commented Jun 6, 2019

Missing log folder

When I try to run nodeindex:build I get an exception:

Exception #1: Warning: file_put_contents(/var/www/html/Data/Logs/ElasticSearch/1559742288.json): failed to open stream: No such file or directory in /var/www/html/Data/Temporary/Development/SubContextDdev/Cache/Code/Flow_Object_Classes/Flowpack_ElasticSearch_ContentRepositoryAdaptor_Indexer_NodeIndexer.php line 424

49 Neos\Flow\Error\ErrorHandler::handleError(2, "file_put_contents(/var/www/html/Data/Logs/ElasticS…: failed to open stream: No such file or directory", "/var/www/html/Data/Temporary/Development/SubContex…h_ContentRepositoryAdaptor_Indexer_NodeIndexer.php", 424, array|7|)
48 file_put_contents("/var/www/html/Data/Logs/ElasticSearch/1559742288.json", "{"index":{"_type":"unstructured","_id":"d55c90ce95…l,"_creationDateTime":"2018-02-23T12:30:57-07:00"}")
47 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::flush()
46 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::flushIfNeeded()
45 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::toBulkRequest(Neos\ContentRepository\Domain\Model\Node, array|0|)
44 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, NULL)
43 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}()
42 Flowpack\ElasticSearch\ContentRepositoryAdaptor\ElasticSearchClient_Original::withDimensions(Closure, array|2|)
41 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, Neos\ContentRepository\Domain\Service\Context)
40 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::indexNode(Neos\ContentRepository\Domain\Model\Node)
39 Neos\ContentRepository\Search\Indexer\NodeIndexingManager_Original::Neos\ContentRepository\Search\Indexer\{closure}()
38 Closure::__invoke()
37 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::withBulkProcessing(Closure)
36 Neos\ContentRepository\Search\Indexer\NodeIndexingManager_Original::flushQueues()
35 Neos\ContentRepository\Search\Indexer\NodeIndexingManager_Original::flushQueuesIfNeeded()
34 Neos\ContentRepository\Search\Indexer\NodeIndexingManager_Original::indexNode(Neos\ContentRepository\Domain\Model\Node)
33 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 100)
32 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
31 array_map(Closure, array|5|)
30 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 100)
29 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
28 array_map(Closure, array|1|)
27 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 100)
26 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
25 array_map(Closure, array|7|)
24 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 100)
23 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
22 array_map(Closure, array|16|)
21 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 100)
20 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
19 array_map(Closure, array|1|)
18 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 100)
17 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
16 array_map(Closure, array|1|)
15 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 100)
14 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::indexWithDimensions("live", array|2|, NULL, Closure)
13 call_user_func_array(array|2|, array|4|)
12 Neos\Flow\ObjectManagement\DependencyInjection\DependencyProxy::__call("indexWithDimensions", array|4|)
11 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Command\NodeIndexCommandController_Original::buildWorkspaceInternalCommand("live", array|2|, "1559742203", NULL)
10 call_user_func_array(array|2|, array|4|)
9 Neos\Flow\Cli\CommandController_Original::callCommandMethod()
8 Neos\Flow\Cli\CommandController_Original::processRequest(Neos\Flow\Cli\Request, Neos\Flow\Cli\Response)
7 Neos\Flow\Mvc\Dispatcher_Original::initiateDispatchLoop(Neos\Flow\Cli\Request, Neos\Flow\Cli\Response)
6 Neos\Flow\Mvc\Dispatcher_Original::dispatch(Neos\Flow\Cli\Request, Neos\Flow\Cli\Response)
5 Neos\Flow\Cli\CommandRequestHandler::Neos\Flow\Cli\{closure}()
4 Closure::__invoke()
3 Neos\Flow\Security\Context_Original::withoutAuthorizationChecks(Closure)
2 Neos\Flow\Cli\CommandRequestHandler::handleRequest()
1 Neos\Flow\Core\Bootstrap::run()

Seems the folder FLOW_PATH_DATA . 'Logs/ElasticSearch/' is not being created autmatically. As a workaround I created the folder.

@DrillSergeant

This comment has been minimized.

Copy link

commented Jun 6, 2019

Undefined index

This one blocks me as I dont have a workaround:

When running nodeindex:build (after quite a while) this happens:

Exception #1 in line 424 of /var/www/html/Data/Temporary/Development/SubContextDdev/Cache/Code/Flow_Object_Classes/Flowpack_ElasticSearch_ContentRepositoryAdaptor_Indexer_NodeIndexer.php: Notice: Undefined index: 37bd35692bda21d808d09ce6d5983b43 in /var/www/html/Data/Temporary/Development/SubContextDdev/Cache/Code/Flow_Object_Classes/Flowpack_ElasticSearch_ContentRepositoryAdaptor_Indexer_NodeIndexer.php line 424

51 Neos\Flow\Error\ErrorHandler::handleError(8, "Undefined index: 37bd35692bda21d808d09ce6d5983b43", "/var/www/html/Data/Temporary/Development/SubContex…h_ContentRepositoryAdaptor_Indexer_NodeIndexer.php", 424, array|9|)
50 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::flush()
49 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::flushIfNeeded()
48 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::toBulkRequest(Neos\ContentRepository\Domain\Model\Node, array|2|)
47 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, NULL)
46 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}()
45 Flowpack\ElasticSearch\ContentRepositoryAdaptor\ElasticSearchClient_Original::withDimensions(Closure, array|2|)
44 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, Neos\ContentRepository\Domain\Service\Context)
43 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::indexNode(Neos\ContentRepository\Domain\Model\Node)
42 Neos\ContentRepository\Search\Indexer\NodeIndexingManager_Original::Neos\ContentRepository\Search\Indexer\{closure}()
41 Closure::__invoke()
40 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\NodeIndexer_Original::withBulkProcessing(Closure)
39 Neos\ContentRepository\Search\Indexer\NodeIndexingManager_Original::flushQueues()
38 Neos\ContentRepository\Search\Indexer\NodeIndexingManager_Original::flushQueuesIfNeeded()
37 Neos\ContentRepository\Search\Indexer\NodeIndexingManager_Original::indexNode(Neos\ContentRepository\Domain\Model\Node)
36 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 706)
35 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
34 array_map(Closure, array|1|)
33 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 706)
32 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
31 array_map(Closure, array|3|)
30 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 706)
29 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
28 array_map(Closure, array|7|)
27 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 706)
26 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
25 array_map(Closure, array|2|)
24 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 706)
23 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
22 array_map(Closure, array|16|)
21 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 706)
20 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
19 array_map(Closure, array|1|)
18 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 706)
17 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node)
16 array_map(Closure, array|1|)
15 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\{closure}(Neos\ContentRepository\Domain\Model\Node, 706)
14 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Indexer\WorkspaceIndexer_Original::indexWithDimensions("live", array|2|, NULL, Closure)
13 call_user_func_array(array|2|, array|4|)
12 Neos\Flow\ObjectManagement\DependencyInjection\DependencyProxy::__call("indexWithDimensions", array|4|)
11 Flowpack\ElasticSearch\ContentRepositoryAdaptor\Command\NodeIndexCommandController_Original::buildWorkspaceInternalCommand("live", array|2|, "1559747524", NULL)
10 call_user_func_array(array|2|, array|4|)
9 Neos\Flow\Cli\CommandController_Original::callCommandMethod()
8 Neos\Flow\Cli\CommandController_Original::processRequest(Neos\Flow\Cli\Request, Neos\Flow\Cli\Response)
7 Neos\Flow\Mvc\Dispatcher_Original::initiateDispatchLoop(Neos\Flow\Cli\Request, Neos\Flow\Cli\Response)
6 Neos\Flow\Mvc\Dispatcher_Original::dispatch(Neos\Flow\Cli\Request, Neos\Flow\Cli\Response)
5 Neos\Flow\Cli\CommandRequestHandler::Neos\Flow\Cli\{closure}()
4 Closure::__invoke()
3 Neos\Flow\Security\Context_Original::withoutAuthorizationChecks(Closure)
2 Neos\Flow\Cli\CommandRequestHandler::handleRequest()
1 Neos\Flow\Core\Bootstrap::run()
@dfeyer

This comment has been minimized.

Copy link
Member Author

commented Jun 6, 2019

As I dont need autocompletion (yet) I could get rid of this problem by disabling the mixin, but the issue may remain if you need autocompletion.

This is not related to this PR, it's a problem with the SearchPlugin and ES5 autocompletion changes, maybe @daniellienert knows more about this ?

@dfeyer

This comment has been minimized.

Copy link
Member Author

commented Jun 6, 2019

Seems the folder FLOW_PATH_DATA . 'Logs/ElasticSearch/' is not being created autmatically. As a workaround I created the folder.

Thanks, last two commit solves the bug and store the request only on errors. Good catch

@dfeyer

This comment has been minimized.

Copy link
Member Author

commented Jun 6, 2019

This one blocks me as I dont have a workaround:

A second really good catch ;) the current bulk request contains nothing to index for a defined dimensions preset, solved by 344d078

@DrillSergeant

This comment has been minimized.

Copy link

commented Jun 7, 2019

Good news: indexing (and searching afterwards) works nicely for me. Thanks for your work. Looking forward to a merge.

@dfeyer One thing I am not sure if this is correct:
When I check the aliases after nodeindex:build I get 3 kinds of aliases:
image
1: Alias named from the setting configured in Neos.Neos.ContentRepository.Search.elasticSearch.indexName
2: Alias named from the setting (see 1) + timestamp of indexing-run
3: Alias named from the setting (see 1) + dimensionhash

Number 3 is needed a correct. But does make those in number 1 and 2 any sense? Even more, they are exisiting multiple times, did not know this is even possible.

@dfeyer

This comment has been minimized.

Copy link
Member Author

commented Jun 19, 2019

When I check the aliases after nodeindex:build I get 3 kinds of aliases:

@DrillSergeant Yes it's not final ... I think we don't need to "meta" alias ;)

I found a bug, when changing the node type, during publishing ES return an error. Can you confirm ? Just to be sure it's not related to my project.

@DrillSergeant

This comment has been minimized.

Copy link

commented Jun 19, 2019

@dfeyer cannot confirm that behaviour. I just changes some document nodes to other nodetypes (page to shortcut => publish, no problem. And back again without problem).
I also changed some content node to other nodetypes without problems on publishing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.