htmllisting-parser

Python parser for Apache/nginx-style HTML directory listing

import htmllistparse
cwd, listing = htmllistparse.fetch_listing(some_url, timeout=30)

# or you can get the url and make a BeautifulSoup yourself, then use
# cwd, listing = htmllistparse.parse(soup)

where cwd is the current directory, listing is a list of FileEntry named tuples:

• name: File name, str. Have a trailing / if it's a directory.
• modified: Last modification time, time.struct_time or None. Timezone is not known.
• size: File size, int or None. May be estimated from the prefix, such as "K", "M".
• description: File description, file type, or any other things found. str as HTML, or None.

Supports:

• Vanilla Apache/nginx/lighttpd/darkhttpd autoindex
• Most <pre>-style index
• Many other <table>-style index
• <ul>-style

Note

Please wrap the functions in a general try... except block. It may throw exceptions unexpectedly.

ReHTTPFS

Reinvented HTTP Filesystem.

• Mounts most HTTP file listings with FUSE.
• Gets directory tree and file stats with less overhead.
• Supports Range requests.
• Supports Keep-Alive.

usage: rehttpfs.py [-h] [-o OPTIONS] [-t TIMEOUT] [-u USER_AGENT] [-v] [-d]
                   url mountpoint

Mount HTML directory listings.

positional arguments:
  url                   URL to mount
  mountpoint            filesystem mount point

optional arguments:
  -h, --help            show this help message and exit
  -o OPTIONS            comma separated FUSE options
  -t TIMEOUT, --timeout TIMEOUT
                        HTTP request timeout
  -u USER_AGENT, --user-agent USER_AGENT
                        HTTP User-Agent
  -v, --verbose         enable debug logging
  -d, --daemon          run in background