Skip to content
Python 3 library for reading and writing warc files
Python
Branch: master
Clone or download
Pull request Compare This branch is 10 commits ahead, 1 commit behind recrm:master.
Latest commit 70aaad5 Aug 13, 2015
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs Ran 2to3, fixed CaseInsensitiveDict Nov 15, 2014
test_data
warc
.gitignore Remove reference to HTTPObject Jul 28, 2015
.travis.yml Integrate with Travis CI. Feb 29, 2012
LICENSE Changed to license to GPL v2 and copyrigh to Internet Archive. Feb 29, 2012
MANIFEST.in Added MANIFEST.in and updated setup.py to inlcude Readme.rst and LICE… Feb 24, 2012
Readme.rst
requirements.txt Added warcscrape.py and supporting files. Dec 15, 2014
setup.py Added some bugfixes Nov 15, 2014
warcscrape.py Added warcscrape.py and supporting files. Dec 15, 2014

Readme.rst

warc3: Python3 library to work with WARC files

Note: This is a fork of the original (now dead) warc repository.

WARC (Web ARChive) is a file format for storing web crawls.

http://bibnum.bnf.fr/WARC/

This warc library makes it very easy to work with WARC files.:

import warc
with warc.open("test.warc") as f:
    for record in f:
        print record['WARC-Target-URI'], record['Content-Length']

Documentation

The documentation of the warc library is available at http://warc.readthedocs.org/.

Apart from the install from pip, which will not work for this warc3 version, the interface as described there is unchanged.

License

This software is licensed under GPL v2. See LICENSE file for details.

Authors

Original Python2 Versions:

  • Anand Chitipothu
  • Noufal Ibrahim

Python3 Port:

  • Ryan Chartier
  • Jan Pieter Bruins Slot
  • Almer S. Tigelaar
You can’t perform that action at this time.