/commoncrawl

Permalink
Switch branches/tags
Nothing to show
Commits on Nov 29, 2017

  1. CommonCrawl -> Common Crawl

    @sebastian-nagel
    sebastian-nagel committed Nov 29, 2017
    c95c4db
Commits on Apr 20, 2017

  1. Merge pull request #13 from bryant1410/master

    @sebastian-nagel
    sebastian-nagel committed Apr 20, 2017 
    Fix broken headings in Markdown files
    8623b0d
Commits on Apr 17, 2017

  1. Fix broken Markdown headings

    @bryant1410
    bryant1410 committed Apr 17, 2017
    a10c059
Commits on Feb 14, 2013

  1. Remove content overflow warning message.

    @ahadrana
    ahadrana committed Feb 14, 2013
    dfd33d1
Commits on Feb 13, 2013

  1. Fix ARCFileReader to deal with payload length mismatch between what is

    @ahadrana
    ahadrana committed Feb 13, 2013 
    specified in the ARC Header and what is actually available in the
decompressed stream.
    ccadbab
Commits on Feb 8, 2013

  1. 1. Add support for ArcFileItem via ARCFileItemInputFormat.

    @ahadrana
    ahadrana committed Feb 8, 2013 
    2. Remove non-maven third party lib dependencies in anticipation of
moving to maven instead of ant.
    eebadae
Commits on Jan 23, 2013

  1. Revert to making ARCFileReader constructor public, and delegating

    @ahadrana
    ahadrana committed Jan 23, 2013 
    S3InputStream swap out in ARCFileInputFormat classes.
    4408d90

  2. Fix grammar errors in README.

    @ahadrana
    ahadrana committed Jan 23, 2013
    56a2a66

  3. Add InputFormats for both mapred and mapreduce packages.

    @ahadrana
    ahadrana committed Jan 23, 2013
    dd58ee7
Commits on Jan 21, 2013

  1. ARCFileReader does not close incoming stream in its close method.

    @ahadrana
    ahadrana committed Jan 21, 2013 
    ARCFileRecordReader needs to assume this responsibility.
    be0f6e3
Commits on Jan 18, 2013

  1. Yet again, remove unused imports.

    @ahadrana
    ahadrana committed Jan 18, 2013
    5f30337

  2. Remove improper HexDump import statement.

    @ahadrana
    ahadrana committed Jan 18, 2013
    97e230b

  3. Fix behavioral issue related to the InputStream provided by the S3N

    @ahadrana
    ahadrana committed Jan 18, 2013 
    FileSystem.
    9719a69

  4. Add simplified command line utility to validate ARC File.

    @ahadrana
    ahadrana committed Jan 18, 2013
    06b9c00
Commits on Jan 17, 2013

  1. Remove all Thrift dependencies from CC project.

    @ahadrana
    ahadrana committed Jan 17, 2013
    0f68264

  2. 1. Fix build.xml to fetch maven ant task properly.

    @ahadrana
    ahadrana committed Jan 17, 2013 
    2. Remove external Hadoop dependency.
3. Add deprecated files back in.
    5e8cf9f

  3. Merge pull request #8 from pshken/master

    @ahadrana
    ahadrana committed Jan 17, 2013 
    Updated "mvn.ant.task.url" URL in build.xml
    1f4f452

  4. 1. Normalize project structure in anticipation of move from ant to mvn.

    @ahadrana
    ahadrana committed Jan 17, 2013 
    2. Deprecated old ARCInputFormat code (moved from hadoop.io to
hadoop.io.deprecated
3. Added new Hadoop FileSystem based ARCInputFormat to hadoop.io package.
4. Added test cases to validate new ARCInputFormat code.
5. ARCInputFormat now returns a BytesWritable buffer as the value type.
   The buffer consists of the HTTP Headers (encoded as UTF-8), followed
   by a trailing CRLF (\r\n).Everything after the trailing CRLF is
   the raw content returned by the original HTTP request.
    546ae8f
Commits on Nov 29, 2012

  1. ant.task URL have updated in build.xml due to dead link

    Shh
    Shh committed Nov 29, 2012
    0858376
Commits on Jun 29, 2012

  1. adding '.gitignore' file. bumped version to 1.0.

    Chris Stephens
    Chris Stephens committed Jun 29, 2012
    d99fcef
Commits on Mar 20, 2012

  1. Added protocol definitions for ArchiveInfo and ParseOutput.

    Ahad Rana Ahad Rana
    Ahad Rana authored and Ahad Rana committed Mar 20, 2012
    a36564f
Commits on Mar 15, 2012

  1. Merge pull request #4 from namin/master

    @ahadrana
    ahadrana committed Mar 15, 2012 
    Fix a S3ServiceException when using an input prefix
    e943209

  2. Do not list directories as resources.

    @namin
    namin committed Mar 15, 2012 
    Listing directories as resources trigger errors such as:
org.jets3t.service.S3ServiceException: S3 GET failed for '/data%2F' XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRange</Code><Message>The requested range is not satisfiable</Message><ActualObjectSize>0</ActualObjectSize><RequestId>F063C5C315CC967B</RequestId><HostId>HiShCYLg5oo+hdZceTVRkhhqebTZL5kl1m2gqf+0a0Mme+CSS0d2e9RERPMmcnPY</HostId><RangeRequested>bytes=0-</RangeRequested></Error>
  at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:416)
  at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:752)
  at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:1601)
  at org.jets3t.service.impl.rest.httpclient.RestS3Service.getObjectImpl(RestS3Service.java:1544)
  at org.jets3t.service.S3Service.getObject(S3Service.java:2072)
  at org.commoncrawl.hadoop.io.JetS3tARCSource.getStream(JetS3tARCSource.java:261)
    e62e186
Commits on Nov 21, 2011

  1. Merge branch 'master' of github.com:commoncrawl/commoncrawl

    @ahadrana
    ahadrana committed Nov 21, 2011
    b835665

  2. Remove accidental check-in of generator timestamp files.

    @ahadrana
    ahadrana committed Nov 21, 2011
    203d719
Commits on Nov 17, 2011

  1. Merge pull request #1 from matpalm/emr

    commoncrawl
    commoncrawl committed Nov 17, 2011 
    slight generalisation so we can build on elastic mapreduce
    24052ae

  2. slight generalisation so we can build on elastic mapreduce

    @matpalm
    matpalm committed Nov 17, 2011
    079ef9e
Commits on Nov 16, 2011

  1. Fix HTML escaping issue in README.md

    @ahadrana
    ahadrana committed Nov 16, 2011
    8f8bd9f

  2. Fix launcher script to classpath build/lib as part of transition to M…

    @ahadrana
    ahadrana committed Nov 16, 2011 
    …aven.
    82c7233

  3. Added introductory README

    @ahadrana
    ahadrana committed Nov 16, 2011
    fa87ea7

  4. Final modifications before recommit to GitHub

    @ahadrana
    ahadrana committed Nov 16, 2011
    1beca4a
Commits on Nov 14, 2011

  1. Fix directory tree.

    Ahad Rana Ahad Rana
    Ahad Rana authored and Ahad Rana committed Nov 14, 2011
    8d2a96d

  2. Minor modification to GoogleURL interface.

    Ahad Rana Ahad Rana
    Ahad Rana authored and Ahad Rana committed Nov 9, 2011
    27243d8

  3. Fixed bug in InputStream implementation.

    Ahad Rana Ahad Rana
    Ahad Rana authored and Ahad Rana committed Nov 9, 2011
    581ff3e

  4. Includes, among other things, (1) added mergeutils project into commo…

    Ahad Rana Ahad Rana
    Ahad Rana authored and Ahad Rana committed Aug 10, 2011 
    …ncrawl source tree (2) added query project into commoncrawl source tree (3) major refactoring of query project (4) bulk scan implementation (5) integration of parallel query functionality (6) bulk query support in cacheFE server (7) fix improper flush bug in Indexer code
    4158d76