Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DS-4194 Use lazy fetching iterator for long query results #2379

Merged
merged 1 commit into from
Mar 29, 2019

Conversation

AndrewZachWood
Copy link
Contributor

Solves: DS-4194

This moves away from reading every result into memory during the Discovery indexing process.

@tdonohue tdonohue added bug high priority component: Discovery Related to Discovery search or browse system labels Mar 21, 2019
@tdonohue tdonohue added this to the 7.0 milestone Mar 21, 2019
@tdonohue
Copy link
Member

NOTE: This only affects master (pre-7.0) as the behavior seems to have changed after we did Hibernate deprecation fixes in #2111. So, this PR seems to be fixing an accidental side-effect of #2111.

Copy link
Contributor

@terrywbrady terrywbrady left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The performance gains are clear even with only 2000 or 4000 items.

2345 Items ======================

$ time winpty docker exec -it dspace //dspace/bin/dspace index-discovery -b -f

real    1m39.683s
user    0m0.000s
sys     0m0.015s

$ time winpty docker exec -it dspace //dspace/bin/dspace index-discovery -b -f

real    2m15.639s
user    0m0.031s
sys     0m0.015s

-----with new code

$ time winpty docker exec -it dspace //dspace/bin/dspace index-discovery -b -f

real    1m16.617s
user    0m0.000s
sys     0m0.046s

$ time winpty docker exec -it dspace //dspace/bin/dspace index-discovery -b -f

real    0m54.334s
user    0m0.015s
sys     0m0.046s

4343 items ===========================================


$ time winpty docker exec -it dspace //dspace/bin/dspace index-discovery -b -f

real    3m40.467s
user    0m0.015s
sys     0m0.046s

$ time winpty docker exec -it dspace //dspace/bin/dspace index-discovery -b -f

real    3m2.371s
user    0m0.000s
sys     0m0.046s


-------- with new code

$ time winpty docker exec -it dspace //dspace/bin/dspace index-discovery -b -f

real    1m27.602s
user    0m0.000s
sys     0m0.046s

$ time winpty docker exec -it dspace //dspace/bin/dspace index-discovery -b -f

real    1m26.785s
user    0m0.000s
sys     0m0.046s

Copy link
Member

@tdonohue tdonohue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the testing, @terrywbrady !

👍 These code changes look good to me. I was just waiting on a verification test.

@tdonohue tdonohue merged commit 1e6497f into DSpace:master Mar 29, 2019
@tdonohue
Copy link
Member

Thanks @AndrewZWood !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug component: Discovery Related to Discovery search or browse system high priority
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants