Improves search to handle smaller search terms.

[medha-14]

Description

Fixes #4734

Type of change

Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist:

• No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
• All tests pass: $ python -m pytest (or $ nox -s tests)
• The documentation builds: $ python -m pytest --doctest-plus src (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ nox -s quick.

Further checks:

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.70%. Comparing base (8235474) to head (5bcd5d0).
Report is 3 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #4735   +/-   ##
========================================
  Coverage    98.70%   98.70%           
========================================
  Files          303      303           
  Lines        23335    23353   +18     
========================================
+ Hits         23032    23050   +18     
  Misses         303      303           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[medha-14]

Could I get a review on this one?

[kratman]

@medha-14 Sorry, there is a bit of a backlog due to the upcoming release and everyone coming back from vacation. Don't worry, we will review this shortly

[medha-14]

Thank you for the update! I just thought you missed this one.

[agriyakhetarpal]

Thanks! I think min_similarity could be slightly higher here because the threshold here is too low and can lead to false positives in the search. But, at the same time, being able to resolve potential typos in the search query would require a lower threshold. This makes me feel that we could expose it with a sensible default, but I don't yet know what a sensible default would be. It should be higher than the current 40%, though. In-line comment about this below:

[medha-14]

Thanks! I think min_similarity could be slightly higher here because the threshold here is too low and can lead to false positives in the search. But, at the same time, being able to resolve potential typos in the search query would require a lower threshold. This makes me feel that we could expose it with a sensible default, but I don't yet know what a sensible default would be. It should be higher than the current 40%, though. In-line comment about this below:

Sorry for the delayed response. I think it's important to clarify that min_similarity is only relevant for substring matches when the search_key is found within the known keys. For cases where there are typos or no substring matches, difflib.get_close_matches() handles those independently, using its own cutoff threshold. Increasing the min_similarity too high would make it difficult to get even the relevant matches.
Eg: If we even set the threshold to 0.5 and a user searches for conc it will not be matched with concentration because it will have similarity ratio of 0.47(approximately) which will not qualify the threshold.

[agriyakhetarpal]

Thanks, @medha-14! Let's go ahead with this for now, I feel your explanation adds the necessary context. I'm happy to approve this once we resolve https://github.com/pybamm-team/PyBaMM/pull/4735/files#r1905523299, as min_similarity is defined in two different locations, and if we have to do that, I think making it an argument that's passed to _find_matches() makes sense, along with documenting it.

[medha-14]

min_similarity is also defined above, so it gets defined twice. How about making it an argument for _find_matches()?

Follow-up question: do you think it would make sense to expose it publicly for users through search() as well?

I’ve made min_similarity an argument for _find_matches() and also exposed it to users via search(), it could be useful to control the threshold according to their needs.

[agriyakhetarpal]

Thanks, @medha-14!

[agriyakhetarpal]

Actually, could you please add a test where min_similarity is used, for the cases where significant_match becomes True or False? It is currently in untested waters.

[medha-14]

I have added appropriate tests that cover cases where we give different values to min_similarity producing different results.

[Saransh-cpp]

Requesting @agriyakhetarpal's review on this.

[agriyakhetarpal]

This is good to merge from my side. I'm mostly waiting on @Saransh-cpp's review as he self-requested one.

[Saransh-cpp]

Woops, sorry for holding this back

[medha-14]

Thanks a lot everyone!