Support for large numbers of features #103
Conversation
There was a problem hiding this comment.
@wasade, looks good, thank you.
Some minor comments. Also, would it be worth adding the batch sizes as global or env variables and defaulting to the current values in the code if not present? This will allow changing that values on the fly for debugging or testing the batch sizes. If you agree and out of the scope of this PR, fine to open as an issue.
Co-authored-by: Antonio Gonzalez <antgonza@gmail.com>
* TST: sample id content type bug * MAINT: fix issue where samples with .raw as a suffix were triggering unexpected returns * Don't suffix twice * Adjsut to account for force of json
| req = s.post(config['hostname'], | ||
| data=_format_request(context, cmd, payload)) | ||
|
|
||
| if verbose: | ||
| print(context, cmd, payload[:100]) |
There was a problem hiding this comment.
Not necessary but have you consider the logging python package?
There was a problem hiding this comment.
Yes, but I would like to consider that out of scope of this PR
|
|
||
| Notes | ||
| ----- | ||
| This method only supports count data. |
|
Sorry, missed the batchsize comment. I don't think it's understood well enough to motivate centralizing. At this point, based on what I know, I suspect that effort will not be for gain. |
1 similar comment
|
Sorry, missed the batchsize comment. I don't think it's understood well enough to motivate centralizing. At this point, based on what I know, I suspect that effort will not be for gain. |
This pull requests expands on the server side logic to support bulk load operations for obtaining indices from identifiers, and the load of identifier specific data.
The motivation is to support tables containing millions of identifiers. When performed individually, these operations require milliseconds per query, and 1 million times 1 millisecond begins to get large. What we're in effect doing here is packing in more data into the individual requests to reduce the HTTP request/response overhead.