[BUG] OSD Saved Objects Import 15* Slower in 2.12+

[BlaiseSaunders]

What is the bug?

OpenSearch Dashboards (OSD) Saved Objects import is now unusably slow for large imports, we went from importing objects in <1s to objects taking consistently between 6s and 15s, quite often 15s every single import now.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Connect to OSD 2.12.0 or 2.13.0 stack
2. Go to the saved objects view
3. Import some objects (NDJSON of any amount of objects, either 1 or 100, either via the GUI or via the API)
4. Time how long it takes, either in the chrome network inspect element or with time curl ...
5. Repeat on OSD 2.11.0 stack and observe time difference

Can import 100 items of NDJSON or a single index pattern as NDJSON, it consistently takes ~15s, with the fastest seen being 6s. On version 2.11 the same tests take under 1s.
Same behaviour can be found on 2.12 and 2.13, 2.11.0 is fine, 2.11.1 untested

What is the expected behavior?
Import happens in <1s

OpenSearch Version
2.12.0, 2.13.0

Dashboards Version
2.12.0, 2.13.0

What is your host/environment?

• OS: Linux
• Version 2.12.0, 2.13.0
• Plugins

Do you have any screenshots?
N/A

Do you have any additional context?
In the testing I have run, I have used LDAP auth between OSD and the cluster nodes. Both on the 2.11.0 and 2.12.0 test environments. 2.11.0 performs as expected, 2.12.0 extremely slow. I have also posted this in the security plugin issues page as I believe may be related auth / auth caching, please advise if this is the wrong place, or if there should only be issue opensearch-project/security-dashboards-plugin#1874

[wbeckler]

Would you be willing to run a test with security off with the same size objects?

Also, could you explain further what kind of objects you are importing?

[ananzh]

@bandinib-amzn could you help here?

[seraphjiang]

looks like we need to build a way to run load test on saved object and

@CCongWang @bandinib-amzn let's catch up next week to discuss this.

[BlaiseSaunders]

My bad on this one actually, poor testing methodology. Turned out the stacks I was deploying with 2.12+ had the default index refresh interval set to 15 seconds which was causing the slow down.... After some better testing managed to isolate it to this.

Hope this helps anyone else that's made this mistake!

[kavilla]

My bad on this one actually, poor testing methodology. Turned out the stacks I was deploying with 2.12+ had the default index refresh interval set to 15 seconds which was causing the slow down.... After some better testing managed to isolate it to this.

Hope this helps anyone else that's made this mistake!

@BlaiseSaunders do you think it's worth to re-open as it's not particularly hard for someone to configure the refresh interval that low. And from checking it seems like the importing doesn't give a good indication of status, just a spinner.