A framework for creating and monitoring scrapers in R(Studio).
A website consists on average of 70 requests, only one will yield your target information. Why would you want to load the other 69?
Transitioning from pre-alpha to alpha version. Therefore, i am open for feedback and contributions. There are still a lot of open issues, but i guess that will always stay that way as the scope is the www.
Currently working on
- transition to new style guide (style_guide.md)
- switching to oo
- removing major bugs
If you are interested to learn more, need help with the setup or get involved, feel free to write me: andreas.liebrand@hof.uni-frankfurt.de. Any message is welcome!
devtools::install_github('Toniiiio/sivis')
Run sivis::chrome_addin_tutorial() for instructions to add the chrome addin.
If the addins do not appear in the "Addins" dropdown list after you have installed sivis, start a new Rstudio Session.
Chrome
In case target data is spread across multliple pages / requests
- choose as many data as possible
- start on the second+ page. (first page data might be in another request)
- check if first page data can be derived from the requests that yield the 2nd, 3rd,.. pages (often the case).