New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only 500 data sets can be downloaded although a collection contains more data sets #1049

Open
NTaller opened this Issue Jul 19, 2018 · 0 comments

Comments

Projects
None yet
4 participants
@NTaller

NTaller commented Jul 19, 2018

Whether a collection contains more than 1000 or 2000 records, only 500 records are downloaded.

@haarli haarli added this to the imeji 4.3 milestone Jul 19, 2018

@haarli haarli added the bug label Jul 26, 2018

@helkv helkv assigned MPDLbrede and unassigned helkv Aug 30, 2018

MPDLbrede added a commit that referenced this issue Sep 3, 2018

#1049 Suspend size limit in ElasticSearch scroll search
Amount of objects that can be retrieved from ElasticSearch was
previously limited to 500. New: no limit.

Changes:
de.mpg.imeji.logic.search.elasticsearch.ElasticSearch:
- Added constant with semantic meaning "get all results/no size limit"
- Added further constants for ElasticSearch (scroll interval, timeout
  value)
- Changed functions searchElasticSearch, searchWithScroll: added
  functionality of searching without size limit.
- re-factored function scrollHitIds
- added function scrollAllResults for scroll search without size limit

de.mpg.imeji.logic.export.FolderExport:
- added constants ElasticSearch.SIZE_GET_ALL_RESULTS and
  ElasticSearch.SEARCH_FROM_START in function retrieveObjects.

MPDLbrede added a commit that referenced this issue Sep 5, 2018

#1049 Suspend size limit in ElasticSearch scroll search
-Re-factored:
Moved search constants up one level in abstraction:
from ElasticSearch to Search interface
removed: ElasticSearch.SIZE_GET_ALL_RESULTS,
         ElasticSearch.SEARCH_FROM_START
added: Search.GET_ALL_RESULTS, Search.SEARCH_FROM_START_INDEX
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment