Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spline UI: search is not working #166

Closed
arvindshmicrosoft opened this issue Mar 26, 2019 · 8 comments

Comments

Projects
None yet
3 participants
@arvindshmicrosoft
Copy link

commented Mar 26, 2019

I am running the UI app JAR locally and while the main functions seem to work correctly, the search function seems to be broken. In that typing in a search string is not returning anything (i.e. the left pane gets blank). I debugged that the REST API call (sample below) is returning a blank array [].

http://localhost:8080/rest/dataset/descriptors?q=mysearchstring&asAtTime=1553556446120&offset=0&size=20

There is no firewall or such - the UI is running local to the browser. From the UI output, there is no exception shown. The only variable is that I am using Azure Cosmos DB Mongo API but again on the server side there is no apparent issue. Is there any further logging I can turn on at the Java side to debug this issue further?

@wajda wajda added the bug label Mar 26, 2019

@wajda wajda added this to the 0.3.next milestone Mar 26, 2019

@wajda

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

What is mysearchstring?
The search text occurrences are only looked up in the following attributes:

  • sparkContext.applicationId
  • sparkContext.appName
  • destination file path
  • Spline Lineage UUID
@algattik

This comment has been minimized.

Copy link
Contributor

commented Mar 26, 2019

The issue is due to a compatibility problem in the Cosmos DB MongoDB emulation.
Running an aggregation query against Cosmos DB:

image

while:

image

In fact the Spline unit tests fail when run against Cosmos DB.

@arvindshmicrosoft

This comment has been minimized.

Copy link
Author

commented Mar 27, 2019

Thank you @algattik for your help isolating this! @wajda is there any way we can change Spline to not use the PCRE search expressions to work around the current limitation in the MongoDB API within Azure Cosmos DB? If there is any way we can change Spline (in a fork) to test the workaround, then we will be glad to try that locally. Just let us know what you might suggest.

@algattik

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2019

I confirm that by changing in BaselineLineageDAO:

  •   val regexMatchOnFieldsCriteria = Seq("appId", "appName", "rootOperation.path") map (_ $regex quote(text) $options "i")
    

to

  •   val regexMatchOnFieldsCriteria = Seq("appId", "appName", "rootOperation.path") map (_ $regex text $options "i")
    

The search works against Cosmos DB.

image

@wajda would you consider any of:

  • disabling quoting in the source code? After all, it can be useful for users to be able to search with regexps
  • replacing the call to Pattern.quote with some code to escape special regexp characters? There's code for doing this circulating around stackoverflow, but I'm not too sure of the quality
  • Otherwise, could we quoting an optional flag, settable through an environment variable?

If you agree to any of those, I'm happy to submit a PR.

@wajda

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2019

@algattik Thank you for your contribution. Absolutely, please feel free to submit a PR. We'll be glad to merge it after review.
Regarding the quoting, I would prefer not to remove it entirely. If your 2nd option works, I'd go for it. The main motivation for having quoting there is to avoid unwanted regex evaluation. E.g. if the user types a path with backslashes, or an app name with brackets or the other special regexp characters.

@wajda

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2019

algattik added a commit to algattik/spline that referenced this issue Apr 3, 2019

Escape special characters one by one when searching Mongo DB, rather …
…than using Pattern.quote(), for Cosmos DB compatibility.

Fixes AbsaOSS#166.
@wajda

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2019

Merged.
Thank you guys, great job!

leborgneromain added a commit that referenced this issue Apr 11, 2019

Cosmos DB compatible search (#182)
* Escape special characters one by one when searching Mongo DB, rather than using Pattern.quote(), for Cosmos DB compatibility.
Fixes #166.

* Added test for quoteSafely

* Moved constant to companion object

* Fixed support for \ and $ chars in search

@wajda wajda closed this in e2b88fd Apr 17, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.