tumblr_backup: Add --save-notes and --cookies options

[cebtenzzre]

I've tested this with only one post so far, but it seems to work alright. It brings in a lot of new code and dependencies, but it's all optional.
This also allows youtube-dl to use the cookies that are provided.

[bbolli]

The and web_crawler seems redundant. We never get here with just save_notes set.

[bbolli]

See line 1074

[bbolli]

I did a very quick review inline. More importantly: is it possible to move the part starting at line 2075 into class WebCrawler and maybe the whole class into a separate file?

[cebtenzzre]

The and web_crawler is so that web_crawler can fail to load (line 1385) and the script can still continue. Now that I think about it, it's probably better if we just throw an exception.

[bbolli]

Yes, better not let the user think the notes were saved when the crawler can't initialize.

[cebtenzzre]

So far I've removed the exception-silencing and moved more code into WebCrawler.

[cebtenzzre]

D'oh, I somehow just realized that cookielib is an official Python module.

[cebtenzzre]

The class is in its own file now. I hope I did that right xD

[bbolli]

You can also move the import checks into the module, then check if an import succeeded with if web_crawler.bs4, e.g. You can also do an additional commit for this, so the evolution can be seen later.

[cebtenzzre]

I'm now trying to figure out how to get it to cleanly exit on SIGINT. Currently it throws a lot of exceptions.

[cebtenzzre]

All of the obvious issues have now been fixed.

[cebtenzzre]

Well, it still seems to (sometimes) either ignore SIGINT, or hang when it's sent. Luckily SIGQUIT (Ctrl+\) still works.

[cebtenzzre]

I've somehow only just realized that JavaScript (and thusly Selenium) isn't necessary for this. In my testing, I must have been modifying too many experimental variables at once.

[cebtenzzre]

Alright, now it actually gets the right notes (I think all of the Selenium requests were simultaneous and on the same virtual tab before c70f09a) and it doesn't eat up an insane amount of CPU anymore. Those are good things.

It's still having issues with SIGINT, and it still doesn't get all of the notes (at least according to the note count -- is that accurate?), but it's good enough for me to use now.

[cebtenzzre]

So, locally I have a version that runs the web crawler as a subprocess. The reasoning behind this is that Python threads aren't real threads, because of the GIL. And we do a lot of work in the web crawler that would probably be better off if it were actually running in parallel (HTML parsing, waiting on HTTP requests, and looping back to get the next set of notes).
I first tried to use IronPython, which doesn't have a GIL, but after building my own AUR package just to get it to build and import modules, I found that it won't support lxml without Ironclad, and Ironclad will only build properly on Windows.

With -k --save-notes -s 200 -n 200 --cookies <cookiefile> on one blog, I compared the execution time of each (note the value of real).

As a module:

real	107.98s
user	98.57s
sys	34.58s

As a subprocess:

real	41.27s
user	71.70s
sys	8.17s

So this is more than a 2x speedup of the overall execution (including everything outside of the crawler). The only downside is that it uses more RAM and CPU, due to running multiple interpreter instances.

Should we make this the default, or perhaps an option?

[cebtenzzre]

This is extremely out of date now. I've been doing a lot of work on a few local forks.
If there's ever further interest in this, I will organize and publish my current code.