fullsync command added, netscape cookie support and more stuff #13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I added an extra command a while ago because i wanted to scrape the metadata for all galleries so i could hammer my own local db without getting banned. Therefor i added a
fullsync
command which will iterate through all galleries pages and filter on id to check if the gallery has already been indexed.For (my own) convenience, i have added the cookiefile package as a dependency so i could easily export cookies from chrome in a netscape formatted file and use that (using
cookies.netscape
file).The new
fullsync
command also uses the newly introduceduriCallInterval
andstartPage
config keys.uriCallInterval
is the sleep timer in second between EACH http call andstartPage
is the offset from which page the scraping should be started, in case the script crashes on page 700 or something.