Create large data set for testing purposes

[bernardbaker]

@un7c0rn I need a large data set for testing purposes.

Can you write a 🐍script that I could run?

The reason for this is because I didn't implement from concurrent.futures import ThreadPoolExecutor as you did in the python script firestore.py.

So we need to run tests to see how well the API processes cope.
And measure usage, per user with different sized data sets.

E.g. 1 x User makes a request of the API which processes 1,000 entries in the Firestore DB.

Some information on the Netlify quota for serverless functions:

• Requests per month: 125,000
• Runtime per month: 100 hours

By default, all serverless functions are deployed with:

• us-east-1 AWS Lambda region
• 1024MB of memory
• 10 second execution limit

I know that Firestore is blazingly fast at processing.
But it would be good to have a dashboard if one isn't already available.

🌞

I think this could help web workers. I'll look at it tomorrow.

[un7c0rn]

@bernardbaker Sounds good can you provide a quick example of the format you need the test data to be in? I'm a little concerned about the Netlify quota. I'm assuming we can scale up requests for a fee as necessary? If we use GCP to hose the JS lambda functions we get 2M / month free: https://cloud.google.com/functions/pricing

That said, I'm also in favor of getting a working end-to-end testable MVP done and worrying about pricing optimizations as a second step.

[bernardbaker]

Hi, I need the data to be in JSON format. Netlify allows us to scale up and down, billing only for what is needed. I think that we'll be ok with Netlify for now after looking GCP pricing. And I agree, let's get the MVP done first. But I do need the test data because I need to assess memory usage and speed of processing. Regards, Bernard T. A. Baker

…

On Fri, Apr 17, 2020 at 7:55 AM un7c0rn ***@***.***> wrote: @bernardbaker <https://github.com/bernardbaker> Sounds good can you provide a quick example of the format you need the test data to be in? I'm a little concerned about the Netlify quota. I'm assuming we can scale up requests for a fee as necessary? If we use GCP to hose the JS lambda functions we get 2M / month free: https://cloud.google.com/functions/pricing That said, I'm also in favor of getting a working end-to-end testable MVP done and worrying about pricing optimizations as a second step. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#46 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADHNTTGZQG6M6C5NCSUT5UTRM74OXANCNFSM4MKC4GUA> .