Feature/2046 1833 improved repository search #2136

ghost · 2019-12-03T15:31:16Z

Large search improvement/documentation/test/refactory.

Contents in at bird sight:

Separation of input parsing logic vs dedicated logic for each search service
Simple search service logic completely re-written
135 search tests (most of which new) in repository-core. Including mixed integration and pure unit tests (the latter on the common parsing mechanism)
Documentation and plenty of javadoc
REST endpoint for sys admins to force re-create the index (rather than just reindex the model) - in addition to previous endpoint and with additional entry in UI form
Search is much leninent to user input, supports multiple tags and wildcards, etc. see docs

Not included:
a. Integration tests for Elasticsearch - those are in the works, but will be added in a different branch

…ocker (image/IDE setup not included) This provides two distinct dev features: 1. A profile that allows persisting the model locally across repository restarts, and simplifies some of the runtime parameters (you only need to provide the github-related personal credentials in the runtime configuration) 2. References to this profile in the ElasticSearchConfiguration, so that is used instead of the "SimpleSearch" when running locally. It is implied that an ElasticSearch instance will run locally on the default ports as single-node (i.e. with Docker). Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

… logs (ugly but better than current) / added todo on other ugly code Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

minimal javadoc even if private / some auto-sorting of imports Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

…_search

1) Delegation of parsing / parameter POJO to external classes for reusal 2) Parsing supports case-insensitive tags and terms 3) All values (but not the tag names) can contain wildcards 4) Only untagged type now supported is name (previously infomodel was supported as well) Not yet supported: 1) Namespace/version search are only supported at service level, but not in indices (so they won't work yet as is) 2) No changes yet in "simple" search Not done yet: 1) Tests 2) More tests 3) And more tests even Global / trivial changes: 1) ModelState enum element names changed to conform with other enums, and allow backwards-resolution of exact type name when searching with no wildcards (this impacts many other classes nominally) 2) Copyrights Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

1. Delegation of parsing / parameter POJO to external classes for reusal 2. Parsing supports case-insensitive tags and terms 3. All values (but not the tag names) can contain wildcards 4. Only untagged type now supported is name (previously infomodel was supported as well) 5. **new** Namespace/version search now supported (models need to be re-indexed for this to work) Not yet supported: 1. No changes yet in "simple" search Not done yet: 1) Tests 2) More tests 3) And more tests even Global / trivial changes: 1) ModelState enum element names changed to conform with other enums, and allow backwards-resolution of exact type name when searching with no wildcards (this impacts many other classes nominally) 2) Copyrights Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

…_search

… from development branch. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

1. Removed useless "expandWildcardsOpen" getter invocation in ElasticSearch (might need to verify whether actually setting the value in settings is necessary) 2. Adjusted search controller to prevent prepending name tag to name-tagged content (which results in name:name:[some name expression] queries) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

1. Improved and moved enum-type normalization to SearchTags enum (from ElasticSearch implementation) for clarity/elegance 2. Updated javadocs to reflect 1. and the fact that enums are indexed as keyword (hence, the normalization that capitalizes the enum's first letter when possible) 3. Minimal logging in SearchTags Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

1. Trivial: * fixed typo constant name in ElasticSearchService * fixed some imports in various classes 2. Small: added persisting local repository profile for testing the simple search, and reference in RepositoryConfiguration 3. Big: refactored simple search to conform with ElasticSearch improvements, namely with multi-tag search, lowercase search, multi-wildcard search and removing non-tagged parsing of anything but name. Impact: * ModelRepository (nominal fix) * SearchParameters (added utilities) * ModelSearchUtil (full rewrite / removed parsing from scope, leaving only service-scoped logic) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

1. Made createQueryFromExpression static 2. Removed useless additional space between WHERE and first constraint when building the query 3. Appended wildcard to the end of names if any in given parameters (and if any applicable), to harmonize with ElasticSearch query and for backwards compatibility (the Angular controller used to do that - moved to the back-end) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

This initial change set includes: 1. Tests for each tag + freetext search for model name 2. Moved the previous tests to the new search package, and modifying a few in cases where the functionality changed 3. Added one test resource 4. Added one intermediary test class for some boilerplate functionality and constants Issues: * Importing models is proving quite time-expensive, and the cost is very salient now that many search tests are added * Due to the architecture in the parent class AbstractIntegrationTest, importing models currently has to take place before each test * As such it would be much better to find a way to import the model statically on @BeforeClass once per test class * However, that may imply some refactory or re-writing Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

1. New SearchTestInfrastructure class to be used statically by search test classes 2. Some additional javadoc 3. Separating original search test into 2 classes by model 4. Refactoring logic in new search classes to leverage the static mechanism Performance gain is huge - now 124 tests run in < 1s total (vs. several minutes previously). Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

1. Added / cleaned up some mixed tests 2. Made SearchTestInfrastructure class final Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

1. Fixed internal invocations of search with untagged type failing in tests (and would fail at least in one occurrence in real life) 2. Extraneous/trivial: fixed javadoc in search service interface Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

…full case-insensitivity in non-wildcarded searches. Also amended some javadoc. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

…rchability for formerly non-text fields such as enumerated types, etc. Added controller and interface definition/ES implementation for force-reindexing all fields according to (assuming changed) mapping. This only works with Elastic Search, just as the other reindex functionality. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

* Created ElasticSearch's Readme.md * Linked to the future public version of the latter in the vorto_repository-1.0.yml resource describing the search API Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

* Better handling of unparsed values in SearchTags, to avoid overloading a search with unwanted garbage in case tag is to be treated as case-insensitive * Better naming of tags in Readme.md Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

* Un-tagged names resolve as before (displayName, description, searchableName) * Tagged names only resolve as displayName (which always seems to equal ModelId#getName) * This only seems to function properly with the Elasticsearch index, and is therefore not supported with the simple search, where both tagged and un-tagged searches will all resolve to [vorto:name], which seems broad enough in hindsight Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

…ndex) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

…less colloquial) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

* Unit tests for common search functionality (i.e. regardless of service logic) Fixes * Tag appending to tagged and un-tagged names containing no wildcards moved from service logic and added automatically to common search logic behind the scenes (the simple search still does convert the wildcards to SQL-friendly, as it should) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

aedelmann · 2019-12-04T01:52:55Z

@mena-bosch

You build fails due to a broken link to Vorto Web Site. U might wanna try https://www.eclipse.org/vorto

ghost · 2019-12-04T07:53:07Z

@mena-bosch

1. You build fails due to a broken link to Vorto Web Site. U might wanna try https://www.eclipse.org/vorto

Hey @aedelmann , thanks for your help!
In hindsight I think this is not related to the protocol and more to the fact that, unbeknownst to me, my IDE decided it would be useful to wrap the copy-pasted link to Vorto home in double parentheses...
Trying again with both https and parenthesis fixed momentarily.

… https instead of plain http 2. Better description of forcing re-create index from UI perspective in doc Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

aedelmann · 2019-12-04T08:56:16Z

Hi @mena-bosch
I created another issue for the Released REST Endpoint Bug: #2137

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

ghost · 2019-12-04T11:52:48Z

Sigh.
Looks like latest merge of #2135 today made my test class' usage of org.eclipse.vorto.repository.domain.User unsupported.
Merging again from dev, fixing problematic invocations and pushing again shortly.

…_search

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

* Local dev setup persisting model locally, and using ElasticSearh on docker (image/IDE setup not included) This provides two distinct dev features: 1. A profile that allows persisting the model locally across repository restarts, and simplifies some of the runtime parameters (you only need to provide the github-related personal credentials in the runtime configuration) 2. References to this profile in the ElasticSearchConfiguration, so that is used instead of the "SimpleSearch" when running locally. It is implied that an ElasticSearch instance will run locally on the default ports as single-node (i.e. with Docker). Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Javadoc on current search implementation. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Partial commit - cosmetics/performance: fixed string concatenation in logs (ugly but better than current) / added todo on other ugly code Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Partial commit - cosmetics: better name for name query method and some minimal javadoc even if private / some auto-sorting of imports Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * ElasticSearch service refactory: 1) Delegation of parsing / parameter POJO to external classes for reusal 2) Parsing supports case-insensitive tags and terms 3) All values (but not the tag names) can contain wildcards 4) Only untagged type now supported is name (previously infomodel was supported as well) Not yet supported: 1) Namespace/version search are only supported at service level, but not in indices (so they won't work yet as is) 2) No changes yet in "simple" search Not done yet: 1) Tests 2) More tests 3) And more tests even Global / trivial changes: 1) ModelState enum element names changed to conform with other enums, and allow backwards-resolution of exact type name when searching with no wildcards (this impacts many other classes nominally) 2) Copyrights Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * ElasticSearch service refactory: 1. Delegation of parsing / parameter POJO to external classes for reusal 2. Parsing supports case-insensitive tags and terms 3. All values (but not the tag names) can contain wildcards 4. Only untagged type now supported is name (previously infomodel was supported as well) 5. **new** Namespace/version search now supported (models need to be re-indexed for this to work) Not yet supported: 1. No changes yet in "simple" search Not done yet: 1) Tests 2) More tests 3) And more tests even Global / trivial changes: 1) ModelState enum element names changed to conform with other enums, and allow backwards-resolution of exact type name when searching with no wildcards (this impacts many other classes nominally) 2) Copyrights Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Removed duplicate repository pom dependency declaration after merging from development branch. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Bugfix: 1. Removed useless "expandWildcardsOpen" getter invocation in ElasticSearch (might need to verify whether actually setting the value in settings is necessary) 2. Adjusted search controller to prevent prepending name tag to name-tagged content (which results in name:name:[some name expression] queries) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Improvement: 1. Improved and moved enum-type normalization to SearchTags enum (from ElasticSearch implementation) for clarity/elegance 2. Updated javadocs to reflect 1. and the fact that enums are indexed as keyword (hence, the normalization that capitalizes the enum's first letter when possible) 3. Minimal logging in SearchTags Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Multiple scopes in commit: 1. Trivial: * fixed typo constant name in ElasticSearchService * fixed some imports in various classes 2. Small: added persisting local repository profile for testing the simple search, and reference in RepositoryConfiguration 3. Big: refactored simple search to conform with ElasticSearch improvements, namely with multi-tag search, lowercase search, multi-wildcard search and removing non-tagged parsing of anything but name. Impact: * ModelRepository (nominal fix) * SearchParameters (added utilities) * ModelSearchUtil (full rewrite / removed parsing from scope, leaving only service-scoped logic) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Small fixes in SimpleSearch business logic: 1. Made createQueryFromExpression static 2. Removed useless additional space between WHERE and first constraint when building the query 3. Appended wildcard to the end of names if any in given parameters (and if any applicable), to harmonize with ElasticSearch query and for backwards compatibility (the Angular controller used to do that - moved to the back-end) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Simple search tests This initial change set includes: 1. Tests for each tag + freetext search for model name 2. Moved the previous tests to the new search package, and modifying a few in cases where the functionality changed 3. Added one test resource 4. Added one intermediary test class for some boilerplate functionality and constants Issues: * Importing models is proving quite time-expensive, and the cost is very salient now that many search tests are added * Due to the architecture in the parent class AbstractIntegrationTest, importing models currently has to take place before each test * As such it would be much better to find a way to import the model statically on @BeforeClass once per test class * However, that may imply some refactory or re-writing Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Search tests: big refactory for performance. 1. New SearchTestInfrastructure class to be used statically by search test classes 2. Some additional javadoc 3. Separating original search test into 2 classes by model 4. Refactoring logic in new search classes to leverage the static mechanism Performance gain is huge - now 124 tests run in < 1s total (vs. several minutes previously). Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Search tests: 1. Added / cleaned up some mixed tests 2. Made SearchTestInfrastructure class final Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Search bugfix: 1. Fixed internal invocations of search with untagged type failing in tests (and would fail at least in one occurrence in real life) 2. Extraneous/trivial: fixed javadoc in search service interface Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Small fixes in ElasticSearch service: removed term queries to ensure full case-insensitivity in non-wildcarded searches. Also amended some javadoc. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Changed ElasticSearch index to reflect wildcard, case-insensitive searchability for formerly non-text fields such as enumerated types, etc. Added controller and interface definition/ES implementation for force-reindexing all fields according to (assuming changed) mapping. This only works with Elastic Search, just as the other reindex functionality. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Documentation * Created ElasticSearch's Readme.md * Linked to the future public version of the latter in the vorto_repository-1.0.yml resource describing the search API Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Small glitches fix * Better handling of unparsed values in SearchTags, to avoid overloading a search with unwanted garbage in case tag is to be treated as case-insensitive * Better naming of tags in Readme.md Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Fine-tune on "name" search resolution: * Un-tagged names resolve as before (displayName, description, searchableName) * Tagged names only resolve as displayName (which always seems to equal ModelId#getName) * This only seems to function properly with the Elasticsearch index, and is therefore not supported with the simple search, where both tagged and un-tagged searches will all resolve to [vorto:name], which seems broad enough in hindsight Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Admin function: re-created index (added instead of just replacing reindex) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Cosmetics: changed "under the hood" to "behind the scenes" (slightly less colloquial) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Tests * Unit tests for common search functionality (i.e. regardless of service logic) Fixes * Tag appending to tagged and un-tagged names containing no wildcards moved from service logic and added automatically to common search logic behind the scenes (the simple search still does convert the wildcards to SQL-friendly, as it should) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Added a couple more search unit tests Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * 1. Build fix: fixed double-parenthesis in link to Vorto home and used https instead of plain http 2. Better description of forcing re-create index from UI perspective in doc Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Last-minute: forgot to fix one of the two URLs in docs Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com> * Another last-minute fix to accommodate signature changes in eclipse#2135 Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

menajrl added 28 commits November 7, 2019 13:35

Javadoc on current search implementation.

a8c616f

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Partial commit - cosmetics/performance: fixed string concatenation in…

ed3a710

… logs (ugly but better than current) / added todo on other ugly code Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Partial commit - cosmetics: better name for name query method and some

bdade30

minimal javadoc even if private / some auto-sorting of imports Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Merge branch 'development' into feature/2046_1833_improved_repository…

a8cea2c

…_search

Merge branch 'development' into feature/2046_1833_improved_repository…

a3dcfeb

…_search

Removed duplicate repository pom dependency declaration after merging…

4020218

… from development branch. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Search tests:

fd749f1

1. Added / cleaned up some mixed tests 2. Made SearchTestInfrastructure class final Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Small fixes in ElasticSearch service: removed term queries to ensure …

70b52f0

…full case-insensitivity in non-wildcarded searches. Also amended some javadoc. Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Merge from latest development branch

67f4e7c

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Documentation

150bb6e

* Created ElasticSearch's Readme.md * Linked to the future public version of the latter in the vorto_repository-1.0.yml resource describing the search API Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Admin function: re-created index (added instead of just replacing rei…

ff1bc99

…ndex) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Cosmetics: changed "under the hood" to "behind the scenes" (slightly …

de80aee

…less colloquial) Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Added a couple more search unit tests

d38c26f

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Merge-in from latest dev

77fd140

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

ghost mentioned this pull request Dec 3, 2019

Integration Tests for Elastic Search functionality in Vorto Repository #2132

Closed

1. Build fix: fixed double-parenthesis in link to Vorto home and used…

31e0bc3

… https instead of plain http 2. Better description of forcing re-create index from UI perspective in doc Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

Last-minute: forgot to fix one of the two URLs in docs

48ad661

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

ghost mentioned this pull request Dec 4, 2019

List of Vorto Function Blocks in Info Model Creation wizard shows all released Function Blocks #2137

Closed

menajrl added 2 commits December 4, 2019 12:53

Merge branch 'development' into feature/2046_1833_improved_repository…

4ee6da6

…_search

Another last-minute fix to accommodate signature changes in eclipse#2135

da3e3f4

Signed-off-by: Menahem Julien Raccah Lisei <menahemjulien.raccahlisei@bosch-si.com>

aedelmann merged commit 74060c9 into eclipse:development Dec 4, 2019

aedelmann deleted the feature/2046_1833_improved_repository_search branch December 4, 2019 13:50

This was referenced Dec 5, 2019

REST calls to the org.eclipse.vorto.repository.web.core.ModelSearchController can be replaced by direct calls to the API search controller #2142

Closed

migration guide for 0.13 #2159

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/2046 1833 improved repository search #2136

Feature/2046 1833 improved repository search #2136

ghost commented Dec 3, 2019

aedelmann commented Dec 4, 2019

ghost commented Dec 4, 2019

aedelmann commented Dec 4, 2019

ghost commented Dec 4, 2019

Feature/2046 1833 improved repository search #2136

Feature/2046 1833 improved repository search #2136

Conversation

ghost commented Dec 3, 2019

aedelmann commented Dec 4, 2019

ghost commented Dec 4, 2019

aedelmann commented Dec 4, 2019

ghost commented Dec 4, 2019