url-meta-scraper is a worker which primary task is downloading URLs that consumed from MQ.
On success it will push the content of URL to queue urls.
If downloading of URL failed, it will push to separate queue failed-urls for further retry by another worker.
- AMQP
WIP
-
Producer for downloaded URLs - Producer for failed URLs
See file LICENSE.