Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

$expand operation needs to be optimized for large ValueSets. #1366

Closed
dmuylwyk opened this issue Jul 2, 2019 · 14 comments

Comments

@dmuylwyk
Copy link
Collaborator

commented Jul 2, 2019

Presently, invoking the $expand operation on large ValueSets takes waaay too long (e.g. try it against a ValueSet that includes the entire LOINC code system). We can do better!

The CodeSystem resource provider stores concepts in separate tables in the database. We should do something similar for ValueSet. This should allow for invocation of the $expand operation on large ValueSets. That's what I'm working on. : )

@dmuylwyk dmuylwyk added the enhancement label Jul 2, 2019

@dmuylwyk dmuylwyk self-assigned this Jul 2, 2019

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 3, 2019

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 5, 2019

Pull request: #1369

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 5, 2019

Next step is to modify the $expand operation to make use of pre-expanded ValueSets in the terminology tables.

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 10, 2019

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 10, 2019

Pull request: #1376

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 16, 2019

Next step is to work on persisting and loading in the background.

We should also persist concept properties when creating/updating CodeSystem resources.

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 29, 2019

@jamesagnew handled persistence of concept properties when creating/updating CodeSystem resources in this commit: a4ca537

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 29, 2019

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 30, 2019

Still need to handle persistence of concept properties to the ValueSet-related terminology tables. See previous comment. We need TermValueSetConceptProperty, similar to TermConceptProperty. Expansions don't include the properties themselves; they are needed to facilitate filters and parameterized expansions. This is to be implemented later (see #1420).

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 31, 2019

Submitted #1403 for review.

Next up: actually use the terminology tables when expanding.

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Aug 8, 2019

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Aug 19, 2019

$validate-code operation needs to be optimized for large ValueSets. See #1431.

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Aug 19, 2019

Pull request: #1432

@dmuylwyk

This comment has been minimized.

Copy link
Collaborator Author

commented Aug 20, 2019

To do:

  • Fix Lucene error; exceeding max clauses. Fixed using BooleanQuery.setMaxClauseCount(10000); Processing slows after any ValueSets with many codes explicitly identified but this might be due to the dark arts that is memory management. Will monitor but not do anything about this right now.
  • Fix deletion of TermValueSets and children. Fixed and optimized for performance.
  • Don't pre-expand ValueSets until all codes have been loaded. Done.
  • Don't break if attempting to $expand during pre-expansion. This is likely limited to Derby.
  • Improved logging. Done (left two TODOs for log messages that mysteriously did not work).
  • Better test coverage. Done (but we could certainly benefit from more).

@dmuylwyk dmuylwyk closed this Aug 26, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.