Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Editoast: add search endpoint #2901

Merged
merged 4 commits into from
Feb 8, 2023
Merged

Editoast: add search endpoint #2901

merged 4 commits into from
Feb 8, 2023

Conversation

leovalais
Copy link
Contributor

@leovalais leovalais commented Jan 16, 2023

Adds the /search?<page>&<page_size> endpoint.

Example payload:

{
    "object": "operationalpoint",
    "query": ["or", 
                ["search", ["name"], "marse"], 
                ["=", ["trigram"], "pno"]],
}

The endpoint is generic enough so that new types of objects can be searched without modifying the route. For example, when we'll add signals into the search engine, we will
create a table named osrd_search_signal with some columns (e.g.: "label", "aspect", etc.). Then we'll update search.yml like so:

operationalpoint: ...
...
signal:
  table: osrd_search_signal
  columns:
    label: string
    aspect: string
    ...
  result:
    joins: ...
    columns:
      obj_id: obj_id
      infra_id: infra_id
      label: data#>>'{extensions,identifier,label}'
      aspect: data#>>'{extensions,identifier,aspect}'
      ...

We will then be able to query the route like this without changing the route's logic:

{
    "object": "signal",
    "query": ["and", 
                ["search", ["label"], "C14"], 
                ["=", ["aspect"], "carre"]
                ["=", ["infra_id"], 2]],
}

Only implemented for operational points for now.

See also #3002
Fixes #1625, fixes #1635

@codecov
Copy link

codecov bot commented Jan 16, 2023

Codecov Report

Merging #2901 (7e13263) into dev (1912f36) will increase coverage by 0.73%.
The diff coverage is 68.88%.

@@             Coverage Diff              @@
##                dev    #2901      +/-   ##
============================================
+ Coverage     38.67%   39.41%   +0.73%     
  Complexity     1709     1709              
============================================
  Files           603      611       +8     
  Lines         18749    19199     +450     
  Branches       2511     2511              
============================================
+ Hits           7252     7567     +315     
- Misses        11091    11226     +135     
  Partials        406      406              
Flag Coverage Δ
editoast 70.46% <68.88%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
editoast/src/main.rs 9.27% <0.00%> (-0.10%) ⬇️
editoast/src/views/mod.rs 94.11% <ø> (ø)
editoast/src/views/search/config.rs 0.00% <0.00%> (ø)
editoast/src/views/search/mod.rs 2.12% <2.12%> (ø)
editoast/src/views/search/dsl.rs 59.43% <59.43%> (ø)
editoast/src/views/search/sqlquery.rs 67.69% <67.69%> (ø)
editoast/src/views/search/typing.rs 82.85% <82.85%> (ø)
editoast/src/views/search/context.rs 83.92% <83.92%> (ø)
editoast/src/views/search/process.rs 92.20% <92.20%> (ø)
editoast/src/views/search/searchast.rs 100.00% <100.00%> (ø)
... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@leovalais leovalais force-pushed the lva/search-back branch 3 times, most recently from 19361b5 to 5ce3eff Compare January 23, 2023 10:37
api/osrd_infra/migrations/0002_infra_search.py Outdated Show resolved Hide resolved
editoast/editoast.yml Outdated Show resolved Hide resolved
editoast/editoast.yml Outdated Show resolved Hide resolved
editoast/editoast.yml Outdated Show resolved Hide resolved
editoast/editoast.yml Outdated Show resolved Hide resolved
editoast/openapi.yaml Outdated Show resolved Hide resolved
editoast/src/config.rs Outdated Show resolved Hide resolved
editoast/src/views/infra/queryexpr.rs Outdated Show resolved Hide resolved
editoast/src/views/infra/search.rs Outdated Show resolved Hide resolved
editoast/src/views/infra/search.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@bloussou bloussou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great ! few things on doc :)

api/osrd_infra/migrations/__init__.py Outdated Show resolved Hide resolved
editoast/src/views/search/mod.rs Outdated Show resolved Hide resolved
editoast/src/views/search/mod.rs Outdated Show resolved Hide resolved
editoast/src/views/search/mod.rs Outdated Show resolved Hide resolved
editoast/src/views/search/queryeval.rs Outdated Show resolved Hide resolved
editoast/src/views/search/queryeval.rs Outdated Show resolved Hide resolved
@leovalais leovalais force-pushed the lva/search-back branch 2 times, most recently from e521a80 to 6bf7a4f Compare January 31, 2023 16:11
init_db.sql Outdated Show resolved Hide resolved
editoast/src/views/search/querytype.rs Outdated Show resolved Hide resolved
@leovalais leovalais marked this pull request as ready for review February 8, 2023 08:35
@leovalais leovalais requested a review from a team as a code owner February 8, 2023 08:35
Uses Full Text Search and pg_trgm Postgres extensions.
@leovalais leovalais force-pushed the lva/search-back branch 3 times, most recently from 78c6c3b to 565f9ab Compare February 8, 2023 15:37
Copy link
Contributor

@bloussou bloussou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has improved a lot ! 🚀

api/osrd_infra/migrations/__init__.py Show resolved Hide resolved
api/osrd_infra/migrations/__init__.py Outdated Show resolved Hide resolved
api/osrd_infra/migrations/__init__.py Show resolved Hide resolved
editoast/Cargo.lock Outdated Show resolved Hide resolved
editoast/src/main.rs Outdated Show resolved Hide resolved
editoast/src/views/search/dsl.rs Outdated Show resolved Hide resolved
editoast/src/views/search/mod.rs Outdated Show resolved Hide resolved
editoast/src/views/search/mod.rs Outdated Show resolved Hide resolved
editoast/src/views/search/mod.rs Outdated Show resolved Hide resolved
api/osrd_infra/migrations/__init__.py Show resolved Hide resolved
leovalais and others added 2 commits February 8, 2023 18:32
editoast:search: fixes sql query binding

editoast:search: add unsafely interpolated data validation

editoast:search: remove pointless generic parameter of QueryEvalEnv

api:search: add GiST index support

editoast:search: add some high-level documentation

editoast:search: clippy

editoast:search: add query validation using editoast.yml + dyn response

editoast:search: ensure the query produces a valid WHERE clause

editoast:search: move pagination in url parameters

editoast:search: fix behavior for null values

editoast:search: return OP's geographic and schematic as GeoJSON

editoast:search: fix migration dependencies

editoast:search: fix init_db.sql formatting

editoast:search: clippy

editoast:search: add impl ApiErrors for search errors

editoast:search: set endpoint to /search

editoast:search: smarter text processing + qualified column names

* Use of FTS text processing features & unaccent extension to make search queries more permissive:
- accents are replaced to their corresponding base letters (é => e, Œ => OE, ...)
- ignores irrelevant characters (apostrophes, signs, line breaks, etc...)
- lexeme ordering to match compound nouns without specifying all their parts & allow 'holes'

* Column names are now qualified by the name of the search table to dismiss eventual ambiguities with joint tables
flomonster
flomonster previously approved these changes Feb 8, 2023
Copy link
Contributor

@flomonster flomonster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GG 🎉

@flomonster flomonster merged commit 80f8980 into dev Feb 8, 2023
@flomonster flomonster deleted the lva/search-back branch February 8, 2023 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Search into OPs Create a generic search engine
3 participants