Covid-API MVP

[andreagrandi]

Hi everyone,

I would like to use this issue to discuss what should be part of the MVP. With MVP I mean the "Minimum Valuable Product" we should try to get out as a first release.

1. Import script: a script that takes each *.csv file available here https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports and import the data in the database. The CREATE or UPDATE criteria will be based on these table keys: Province/State, Country/Region and Last Update.

2. Have a few (we will decide them in a separate issue and link them here) endpoints offering basic data queries

3. Deploy the project to Heroku and make it live

4. Add a few usage examples to the README

After these points are done, we should try to advertise the service as much as possible and base the future priorities on the feedback we receive from people using it.

What do you think about? cc @MatMoore @andreagrandi @fundor333

[MatMoore]

Sounds good to me. Is there anything else we can do to encourage feedback?

[andreagrandi]

Sounds good to me. Is there anything else we can do to encourage feedback?

I would be tempted to try and share the project on Hacker News (yeah... I know....) but I'm really not sure if we should try to boost it now that we are in the development phase or later when we release the first version. Any thoughts?

[MatMoore]

I think a good time would be as soon as we get something on heroku that is up to date (even if it's presented as a demo rather than the actual thing). That way people can try it out and hopefully give us more relevant feedback.

Btw, I just noticed that the root URL still returns a 404 which is probably going to be confusing. Can we redirect that to the docs, or put in an HTML landing page?

[andreagrandi]

I think a good time would be as soon as we get something on heroku that is up to date (even if it's presented as a demo rather than the actual thing). That way people can try it out and hopefully give us more relevant feedback.

I agree with you. Also: before advertising the service too much, I would definitely try to get in touch with Heroku folks and literally "beg" for a few credits or, if the service is successful, we won't be able to serve everyone 😓

Btw, I just noticed that the root URL still returns a 404 which is probably going to be confusing. Can we redirect that to the docs, or put in an HTML landing page?

I totally agree on this. Should we create a separate Issue/PR maybe? But I have no idea how, for example, point to the docs. Maybe a static landing page would be better so we can customise it as much as we want.

[MatMoore]

Cool, I've created #22 to continue that discussion.

Good thinking on asking for credits! I think we should look at CDNs as well to protect the heroku servers from increased traffic. I haven't used it before, but it seems like Cloudfare have some sort of free tier we could use: https://www.cloudflare.com/plans/

[andreagrandi]

Cool, I've created #22 to continue that discussion.

Good thinking on asking for credits! I think we should look at CDNs as well to protect the heroku servers from increased traffic. I haven't used it before, but it seems like Cloudfare have some sort of free tier we could use: https://www.cloudflare.com/plans/

I use CloudFlare for my personal blog already and it's quite nice (I survived a few hours in the top ten of HN once, so I think it's good 😅), but it's a static website.

How does a CDN work in case of urls with parameters? ie: you can cache /api/v1/daily-reports/today/ but can you cache something like /api/v1/daily-reports/?country='italy' ?

[MatMoore]

I think the default behaviour treats query parameters as part of the URL, based on this page: https://support.cloudflare.com/hc/en-us/articles/200168256-Understand-Cloudflare-Caching-Level

[andreagrandi]

I think the default behaviour treats query parameters as part of the URL, based on this page: https://support.cloudflare.com/hc/en-us/articles/200168256-Understand-Cloudflare-Caching-Level

this is interesting! Do we need a custom domain? Should we get one anyway? Shielding the Heroku instance is something I would definitely like to do, but we don't know yet if and how many people will use the API.
Do you think it could be possibile to initially stick with Heroku only? Or could it be a problem to change the url in the future?

[MatMoore]

Ah yeah, I think we'd need a custom domain to use it. But if we do get a custom domain then we'll have the flexibility to change where it points to any time without users having to update URLs.

I think it's fine to stick with heroku initially but if the url is likely to change in future we can make that clear to users in the documentation.

[andreagrandi]

@MatMoore do you think I should create separate Issues for point 1 and 2?

[MatMoore]

Yeah that'd be helpful just so it's clearer what's being worked on. I can pick one of those up monday evening.

[andreagrandi]

Yeah that'd be helpful just so it's clearer what's being worked on. I can pick one of those up monday evening.

I created this for now, which seems to me the most urgent #24

I will take care of creating the account on Heroku and try to ask for credits.

[Alig1493]

is the project still up and running? I could lend a hand with a few things if it's open for public contribution.

[MatMoore]

Hi @Alig1493, we haven't worked on this for a while, and the API is not running anymore, so you might be better off forking or contributing to another project. If you find a good replacement maybe we can link to it in the readme and archive the repo.

Since the available data has changed throughout the pandemic I expect that the way we process data from Johns Hopkins CSSE is out of date now.

[Alig1493]

I see. well the data repo from where you get the data is still. getting regular updates even now in case that information helps in any way.