Is it OK to use /api/0.6/notes/search with large limits?

[matkoniecz]

I am not sure whatever in ENT8R/NotesReview#1 I worry without a good reason or is it a good idea to reduce default limit to avoid overloading API.

I am also not certain is it the best place to ask, but I was unaware about a better one.

[zerebubuth]

10,000 does seem like a lot! But that's the limit set in the source code, and I've no reason to expect it wouldn't be OK.

Perhaps give it at a go against the development / testing server and see if it has acceptable performance there? If we see that there are problems, we might need to reduce it at a later date.

[matkoniecz]

Perhaps give it at a go against the development / testing server and see if it has acceptable performance there?

Is my interpretation "development / testing server has lower performance and potential problems will be easier to notice, queries with decent performance on dev server are likely to not use too many resources" correct?

[zerebubuth]

Is my interpretation "development / testing server has lower performance and potential problems will be easier to notice, queries with decent performance on dev server are likely to not use too many resources" correct?

It's certainly the case that if the performance is terrible then we'll notice it on the dev server, and might be able to fix it before hurting the main production site.

But it's not always the case that things which run well on the dev server will run well on the production site, as the dev server has a tiny fraction of the data that the production site has, and runs on completely different hardware. But the good news is that we can try and fake it by adding a whole bunch of notes on the dev server to test how it scales.

Another possibility is that we could add a better interface for dealing with note data in bulk. Would a "note changes stream" similar to the minutely diffs be useful?

[pnorman]

Another possibility is that we could add a better interface for dealing with note data in bulk. Would a "note changes stream" similar to the minutely diffs be useful?

Yes. People currently fake this by fetching note n+1 and pausing when it fails, but this is hackish, fails when the serial goes up by more than one, and isn't a great design.

I don't know if a notes stream helps this particular use-case, but there are others it would.

[pnorman]

Question was answered, and although a new notes change stream would be nice, this issue isn't the right place for that to develop.