remote-work

crawl and download files from an open-directory

npm install remote-work

**work in progress!**

Sometimes you'll open a webpage, and it will look like this:

This is called an open directory, or sometimes an autoindexer.

It's a server that's configured to show you all its files, which is nice. It used to be more common.

This is a tool to download the all files from a page like this, from the command-line.

npx remote-work http://us.archive.ubuntu.com/ubuntu/pool/multiverse/y

(you'll need to have NodeJS installed)

Features

• async - downloads files 3 at a time, by default
• configurable - download only the files you'd like, using a glob
• stoppable - gets files depth-first
• resumable - don't re-download files that you already have

Node API

you can also use this library in a script npm install remote-work

import remoteWork from 'remote-work'

const url = 'http://us.archive.ubuntu.com/ubuntu/pool/multiverse/y'
const dir = './output'
let opts = {
  n: 1, //only download one file at a time
  match: '*.mp3' //only download mp3 files
}
await remoteWork(url, dir, opts)

Please be considerate when downloading files from a remote server.

---

See also

• wget-wizard - do it all w/ a CLI script
• reddit.com/r/opendirectories
• directory_downloader - python directory parser/crawler
• autoindex javascript open directory parser by John Weis

MIT