A downloader / parser to fetch videos with meta data from railscasts.com with support for pro accout. It does two things: 1. fetch all the video you could access 2. mirror an simple rails-casts server
- Let the page parser to talk with the model
The speed within domestic China is too slow for smooth streaming from railscasts.com, so I decided to create a parser/downloader to fetch the videos all at once with their individual meta data.
superagent is used to setup cookie token to access my professional account and send request.
jsdom is used to parse the HTML file and fetch meta data.
kue is used to store and process download tasks in a job queue.
express is used to simulate a simple web server.
mongoskin is used as the backend model driver.
$RC_TOKEN: to hold your login token which will be used to login your professional account.
$RC_ACCOUNT is used to hold HTTP Auth infomation:
$MONGOSKIN_URL is the info used by the app to connect with mongodb, it follows the format in the doc
Install dependencies with:
Need to have redis server installed to use
kue, communicate with the app through the
$REDISTOGO_URL variable in the format
npm run fetch to start to fetch to all assets url.
npm run process to start download all assets.
npm run import to import data to mongodb
npm start to start the server