add(Soopat.js): New translator for a patent site www.soopat.com #3007
Conversation
| if (selectedItems) { | ||
| var urls = Object.keys(selectedItems); | ||
| Z.debug(urls[0]); | ||
| await Promise.all( |
There was a problem hiding this comment.
This will initiate the requests asynchronously, which in practice means almost simultaneously. Some sites don't like this (because of the botty behaviour). Is this perhaps related to the Captcha issue?
There was a problem hiding this comment.
You are right, frequent requests will trigger Captcha for free users. It's not recommended to scrape many items from the search page. How about get page on by one in a for loop.
if (selectedItems) {
var urls = Object.keys(selectedItems);
for (let url in selectedItems) {
let html = await requestDocument(url);
await scrape(html, url, loginStatus);
}
}
There was a problem hiding this comment.
Yeah, we almost always want to do await instead of Promise.all() for this reason. (And if that's not enough, we can consider artificial delays.)
There was a problem hiding this comment.
The Soopat seems to have a strict policy against web scraping for free users. Sometimes I have to enter the Captcha code just after I refresh the webpage. I think there is little thing we can do on Zotero's side.
Soopat.com is a patent database based in China that is widely used by Chinese users. It collects patents from all around the world, not just from China, and allows users to search and download patent applications and authorization PDFs. Pro users can access more detailed data on this site.
The translator extracts metadata from HTML elements rather than using an API. The data in HTML is well-formatted and easy to read.
Please be aware that the pro version requires an account to access. I was given a private account to create this translator.