refine_scopus - low it/s speed; necessary to refine every time?

[FabianEUR]

Hi,

Thanks for developing this package and the example/guide. Extremely useful for someone with basic python skills. From the example, I saw that you were managing to get very high it/s speeds when calling the refine_scopus function. On my end, I'm getting max 1.5it/s when running the function for the first time, both on campus network and via university VPN remotely, which makes an initial run take very long. When I make small changes to my code, such as plot or topic changes and run it again, it refines the documents again, however much quicker.

1. Is the low speed due to something on Scopus' end if my regular internet speeds are very high or is there something I need to include in the code to access higher speeds? I've more or less followed the litstudy example code step-by-step.

2. Is there a way to skip the refine_scopus after running it the first time, i.e., keeping the refined docs locally? I've tried removing the function on a 2nd run and the results for topic, top words, etc. were slightly different than when letting refine_scopus run.

Happy to provide more information/clarify.

Thanks in advance

[stijnh]

Hi, thanks for using using litstudy and thanks for reporting this issue!

Unfortunately, the Scopus API has a limit on how many requests can be performed per minute. The first time that you refine a document set, it can take a long time unfortunately (I typically get 1 request per second).

However, after refining once, the Scopus results should be stored locally on you computer, so refinement should be faster after the first time (it might still take few seconds though).

I would recommend working in a Jupyter notebook. There you can work interactively and make changes to the code while the documents are kept loaded all the time.