increase section experiment limit to 2K

[davidsbailey]

see slack. we are running into this limit while trying to add users to the ai-rubrics experiment. we have 1000 of these objects in production today, so unless there are signs of performance problems, I'd like to propose increasing the limit.

[sureshc]

During peak utilization (5AM PDT to 5PM PDT) our web application is issuing about 10 SELECT statements per second on the experiments table and is retrieving about 4500 rows per query with an average latency of ~5ms (10X slower than our overall SELECT latency). These queries are somewhat intensive ... they retrieve all active experiments and then inspect the result set within the web application server to determine which experiments apply to the current user session. Increasing the number of experiments will increase the size of these result sets, but likely will not degrade overall database performance.

These data is from Performance Insights on the writer database instance

NewRelic confirms that ActiveRecord Experiment find is being executed about 10 times per second during peak usage. The latency from the web application server is 10ms.

Several different Rails controller actions are invoking this ActiveRecord find method. I haven't looked at how they invoke the Experiment model, but we must be caching the result set in memory because our SELECT throughput is very low (10/second) for something that has to be checked several thousand times per minute across a wide variety of high throughput controller actions:

[sureshc]

Our application is inefficient when querying this data, but is likely caching the result set, so increasing the size of the result set should not significantly degrade database system performance.