Handle large user set #1800
Handle large user set #1800
Conversation
nathandunn
commented
Nov 8, 2017
•
nathandunn
commented
- handle searching
- handle sorting
- retest editing
ghost
added
the
|
@deepakunni3 This is ready for testing / review. |
|
@nathandunn what do you consider a large user set? 500, 1,000, 10,000? |
|
This use-case if for 1000 users, each with an individual organism. I know you’ll have several hundred, so it will probably be worth it to you at some point to you.
I haven’t seen any for only a hundred, though.
Nathan
… On Nov 8, 2017, at 8:25 AM, childers ***@***.***> wrote:
@nathandunn <https://github.com/nathandunn> what do you consider a large user set? 500, 1,000, 10,000?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#1800 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAt2qvinpb29jpeDVH-qCysy-3u7sg41ks5s0bnKgaJpZM4QV5w0>.
|
|
first impression (with 1000 users and 1001 organisms):
|
|
Can you provide some details on the API being slow?
Using your data dump it seems pretty snappy, but I just used the interface (which uses the API as well).
Nathan
… On Nov 8, 2017, at 9:41 AM, Michael Paulini ***@***.***> wrote:
first impression:
the rest API to add users is still very slow
the web interface is working and paginating properly without any problems
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#1800 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAt2qibDS9ZQWr9K7cwMYx_m3iD_ER8cks5s0cvDgaJpZM4QV5w0>.
|
|
@nathandunn Tested and looks good. 👍 |
|
Okay. I did it with the very large database and same result. We can address that as a separate issue.
Nathan
… On Nov 8, 2017, at 10:34 AM, Deepak ***@***.***> wrote:
@nathandunn Tested and looks good. 👍
Didn't have any issues with the API, but that could be because I am adding users to a somewhat fresh database.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
We are making 3 specific call to the API for each user:
With this PR, we are finding (1) to now be fast. However, (2) is very slow, and we are getting timeouts. Could it be that the re-engineering you've done for Users also needs to be done for Organisms? Just a thought. |
|
That is the most likely scenario. I’ll take a look.
Nathan
… On Nov 8, 2017, at 10:34 AM, Deepak ***@***.***> wrote:
@nathandunn Tested and looks good. 👍
Didn't have any issues with the API, but that could be because I am adding users to a somewhat fresh database.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
@khowe @epaule with the PR #1801 if you call organism/addOrganisms add the flag: "returnAllOrganisms:false" and it should speed things up quite a bit (otherwise it becomes a very LARGE transaction). When you call "findAllOrganisms" you should be able to (already) provide the an "organism" key / value flag to filter out the return using the commonName or the ID. |
|
Thanks @nathandunn . We're already using the organism key with findAllOrganism, but the returnAllOrganisms:false flag on addOrganism will certainly help! Would there be any chance of adding that parameter for user/createUser too? |
|
@khowe user/createUser should not have this problem (unless you want to add the option to add the full user list with a flag). |
|
Ah yes @nathandunn , you're right. I was actually thinking of user/loadUsers. You can currently restrict the response to a single user with the userId parameter. But it would be useful for us to have an additional parameter to make it only include an organism in the organismPermissions array if the organism has non-empty permissions. e.g. "omitEmptyOrganisms:true" (or something like that). |
|
That seems pretty reasonable.
Nathan
… On Nov 8, 2017, at 3:41 PM, Kevin Howe ***@***.***> wrote:
Ah yes @nathandunn <https://github.com/nathandunn> , you're right.
I was actually thinking of user/loadUsers. You can currently restrict the response to a single user with the userId parameter. But it would be useful for us to have an additional parameter to make it only include an organism in the organismPermissions array if the organism has non-empty permissions. e.g. "omitEmptyOrganisms:true" (or something like that).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#1800 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAt2qlKK15D0DGsXCruBayLxm_3XwelUks5s0iASgaJpZM4QV5w0>.
|
|
Okay. I added a PR for that.
… On Nov 8, 2017, at 3:41 PM, Kevin Howe ***@***.***> wrote:
omitEmptyOrganisms
|
