Write an example of how to get aaall the data via Athena, not the API

[maelle]

The API currently provides access only to the latest 90 days.

I think this package should still focus on the API but it'd be good to write a vignette / a doc that's not a vignette (for the pkgdown website) to illustrate how to get more data from R.

• Using @jflasher's gist https://gist.github.com/jflasher/573525aff9a5d8a966e5718272ceb25a

• And using @hrbrmstr's metis package https://github.com/hrbrmstr/metis -- so mostly a translation/wrapping of Joe's code into R.

Bob, Joe, I have mentioned you two so that if you meet an R user that'd like to write this doc I'm mentioning you send this person to me 😜 Otherwise I'll write it myself in the medium term.

[jflasher]

cc @rickpeltier as I believe he uses this library and has been looking at Athena of late and may be interested in this issue.

[hrbrmstr]

I can offer up any info needed from my daily use of Athena for work-work (we've got quite a bit of data in it and make extensive but straightforward use of partitioning to save $ on queries). Also, def let me know if you need any more helper functions in the package. It "Does What I Need It To Do™" so it doesn't get much dev💙 from me (I rly needed it for the result set size and the temp cred support).

Having said that, https://github.com/nfultz/AWR.Athena is on CRAN and @nfultz and @daroczig are also top-notch coders and likely use Athena in different ways than I do so it could be good to get a few perspective in.

[maelle]

Thanks a lot Bob, I had no idea there was another package, somehow I only found yours via Google! I shall have a look at both then, when I do take the time to wrap Joe's code.

[maelle]

https://gist.github.com/jflasher/573525aff9a5d8a966e5718272ceb25a

[nfultz]

RStudio also supports Athena in their professional drivers package, for whatever that's worth.

[maelle]

I used AWR.Athena to get OpenAQ data via Athena and will report a bit on this in a talk next week. Slides. Webinar, March the 31th, 11:AM EST. Thanks a lot to AWR.Athena authors. 🙂

Now, not sure when I'll have time to report on this in an article, but hopefully soon...

[jflasher]

Cool to see this! @maelle, openaq/openaq-fetch#521 may be of interest. Basically, we're looking to move to a different storage mechanism for the data which would make the queries run much more quickly with almost no change on the Athena side.

cc/ @sruti @RocketD0g

[maelle]

Thanks!

Another R package to interact with Athena https://github.com/DyfanJones/noctua