Feature request: advanced search interface

[keara-soloway]

Even novice FOXDEN / CHESS users who don't know MongoDB query syntax will need to be able to construct queries that involve more complicated query expressions. For example:

• use $all and $size to match an array-valued field, but ignore the order of elements in that field
• use $lt / $gt to match values of a field within a certain range
• use $or and / or $and to match related records from different beamtimes
• ...

The feature request is: an "advanced search" page that presents the user with HTML elements that can be used to construct a query which requires more complicated query expressions. What I have in mind is something similar to the Query Builder page from the REACH ticketing system that CHESS uses:

I imagine the advanced query form would need to be generated from each metadata record schema -- it's still unclear to me whether or not users will need the option to perform advanced searches involving records with different schemas.

[keara-soloway]

@jpk96

[vkuznet]

@keara-soloway , @jpk96 I don't want to jump developing complex web UI until I head those complex queries examples. Any design decision should be based on concrete use-cases, and only once we understand them we can come up with something would it be web UI or CLI or something else. We already have 157 unique keys in different schemas and I think that web form with more than 10 entries is useful as its complexity usually overshadow its practical use. Instead, I rather prefer to see AI driven approach from human readable question translated into specific queries (but that a much broader task).

[vkuznet]

After some thoughts I implemented advanced search interface which you can try out over here:
https://foxden-dev.classe.cornell.edu:8344/advancedsearch

The query itself is not tested/working but web UI to build FOXDEN query is fully functional. I would appreciated if @keara-soloway and @jpk96 will try it out and let me know your feedback.

[jpk96]

@vkuznet Thank you for implementing this quickly! I have confirmed that selecting a schema correctly updates the key selector dropdown menu. Aside from the "Clear" button, none of the other buttons appear to have any effect. As you mentioned, the query functionality doesn’t seem to be working now. Clicking "Search" simply displays all records I have access to.

[vkuznet]

Peter, I added missing search functionality and now we can do the advanced search. For instance, I was able to compose this query using new interface:

And, when I hit search button with this query it correctly looked up metadata record.

Could you please try it out and place some queries on your own to see if functionality is working.

[jpk96]

@vkuznet I ran several queries, and overall they are working as expected. However, I noticed a few points worth mentioning:

1. Operators "equal" and "regular expression" result in functionally identical queries:

# using equal
{
    "$and": [
        {
            "technique": {
                "$regex": "WAXS",
                "$options": "i"
            }
        }
    ]
}


# using regular expression
{
    "$and": [
        {
            "technique": {
                "$options": "i",
                "$regex": "WAXS"
            }
        }
    ]
}

2. Typo in operator names: "greater than" and "less than".

3. Behavior of "greater than" and "less than": These operators are intended for numeric attributes, but currently they return no results. For example, using the ID3B schema and searching for beam_energy less than 9.8 produces the following query but does not return any results.

{
    "$and": [
        {
            "beam_energy": {
                "$options": "i",
                "$lt": 9.8
            }
        }
    ]
}

Removing "$options": "i" from the query returns the expected results.

4. Operator dropdown menu usability: It would be helpful if the operator dropdown only displayed applicable options. For instance:

• If a key’s value is a string, options like "greater than" or "less than" could be hidden.
• If a key’s value is numeric, "regular expression" may not be relevant.

[vkuznet]

@jpk96 , thank you for testing and reporting on this. Let me reply for each item:

1. the equal condition is alike as regex because it is the only way to make it case insensitive. I can change it to = but in that case the search would be case sensitive and I doubt we want that. Please weight on it with your opinion
2. done
3. done
4. I need more time on this as it requires to propagate entire schema to web UI (rather the keys I've done so far). As such the code will be more complex but I think it it is still doable.

So, please let me know what do you think about case-sensitive vs case-insensitive search when equal condition applies.

[vkuznet]

@jpk96 , I encourage you to look at new FOXDEN AIChat capabilities. Just navigate to search page and click on AI button and then ask questions about FOXDEN.

Meanwhile, I'm posting here a working AIChat response to my question about FOXDEN queries to demonstrate its current capabilities:

Werner suggested that we may complement it with a search button which will take autogenerated query and find FOXDEN data. There are severals aspects we should clarify before doing AI based approach:

• deploy AI services into production
• we need more documentation about FOXDEN queries, APIs and MongoDB syntax
• we need to test accuracy of returned autogenerated query results
• we need to add proper scope value for AI queries and handle their validation and sanitization in code

Once, all of the above criteria will be satisfied we can adjust AI chatbot accordingly and put it into production

[jpk96]

@vkuznet The link to the advanced search interface no longer appears on the search page of the dev instance. Navigating directly to https://foxden-dev.classe.cornell.edu:8344/advancedsearch results in a 404 Page Not Found error. Clicking on the "AI" icon does not do anything.

[vkuznet]

@jpk96 , Peter I brought up to live dev instance again. We have some misconfiguration issue with system restarts which I and Devin should figure out. Feel free to try it again and I'll keep eye on it that it should be functional.

[jpk96]

@vkuznet Thank you! Both "advanced search" interface and the AI button are working on the dev instance now.