Skip to content

/ webarchive-indexing

forked from ikreymer/webarchive-indexing
Permalink
Branch: master
Commits on Jun 19, 2019

  1. updating job to index CDXs with python bin path

    root
    root committed Jun 19, 2019
    1566b46
Commits on May 23, 2019

  1. updating code to run on python3.5

    @Fernando-Melo root
    Fernando-Melo authored and root committed May 23, 2019
    194a0e5
Commits on May 21, 2019

  1. Update IndexArcs

    @igobranco root
    igobranco authored and root committed May 21, 2019
    a3df6be

  2. Update gitignore.

    @igobranco root
    igobranco authored and root committed May 21, 2019
    91598b5

  3. Update RunhadoopJobs.sh to use python3.5

    @igobranco
    igobranco committed May 21, 2019
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    efad047
Commits on May 20, 2019

  1. Update hadoop version

    @igobranco
    igobranco committed May 20, 2019
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    4064567

  2. Freeze pywb version

    @igobranco
    igobranco committed May 20, 2019
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    b41143c
Commits on Jun 27, 2018

  1. Now only indexing cdx requests that correspond to http status codes o…

    @Fernando-Melo
    Fernando-Melo committed Jun 27, 2018 
    …f 100s, 200s and 300s. Changed hadoop input type to lineinput to run faster in the new cluster
    c5055a2
Commits on Dec 12, 2016

  1. Utilitie to convert CDX to CDXJ format

    @danielbicho
    danielbicho committed Dec 12, 2016
    49defd8
Commits on Dec 5, 2016

  1. index arcs to cdx

    @Fernando-Melo
    Fernando-Melo committed Dec 5, 2016
    9d1b3de

  2. Index Arcs to CDX

    root
    root committed Dec 5, 2016
    bbc2547
Commits on Feb 22, 2016

  1. indexwarcsjob.py: update hard-coded path to be correct (TODO: load fr…

    @ikreymer
    ikreymer committed Feb 22, 2016 
    …om env)

reqs: update to correct python-hadoop git
    0ee5018
Commits on Nov 24, 2015

  1. add newline after last line in zipnum block, fixes ikreymer#4

    @ikreymer
    ikreymer committed Nov 24, 2015
    7041826
Commits on Oct 9, 2015

  1. build local zipnum: add line numbering to final index (summary), add …

    @ikreymer
    ikreymer committed Oct 9, 2015 
    …option to specify cdx lines per block
    b0ad9c3
Commits on Jun 11, 2015

  1. Merge pull request ikreymer#2 from machawk1/patch-1

    @ikreymer
    ikreymer committed Jun 11, 2015 
    Fixed various MD weirdnesses
    9e90b67

  2. Fixed various MD weirdnesses

    @machawk1
    machawk1 committed Jun 11, 2015
    6edfded
Commits on Mar 29, 2015

  1. minor fixes from latest CC index build:

    @ikreymer
    ikreymer committed Mar 29, 2015 
    - fix typos
- update reducer start for cluster job to be much later
- update reqs
    0005813
Commits on Mar 13, 2015

  1. readme tweaks

    @ikreymer
    ikreymer committed Mar 13, 2015
    70339a0

  2. readme tweaks

    @ikreymer
    ikreymer committed Mar 13, 2015
    e145669

  3. Add local cluster build info

    @ikreymer
    ikreymer committed Mar 13, 2015
    35b85c4
Commits on Mar 12, 2015

  1. Merge branch 'master' of https://github.com/ikreymer/webarchive-indexing

    @ikreymer
    ikreymer committed Mar 12, 2015
    aae924d

  2. add local zip num cluster building support!

    @ikreymer
    ikreymer committed Mar 12, 2015
    fc6223b
Commits on Mar 11, 2015

  1. sample job: add scaler to mapper also

    @ikreymer
    ikreymer committed Mar 11, 2015
    6bee704

  2. readme update

    @ikreymer
    ikreymer committed Mar 11, 2015
    b25e634

  3. Update README.md with more usage instuctions.

    @ikreymer
    ikreymer committed Mar 11, 2015
    70c255a

  4. make scripts runnable

    @ikreymer
    ikreymer committed Mar 11, 2015
    89ab48c

  5. update path to index_env.sh

    @ikreymer
    ikreymer committed Mar 11, 2015
    4f5e4db

  6. add sample index_env.sample.sh

    @ikreymer
    ikreymer committed Mar 11, 2015
    f079e8a

  7. fix typos

    @ikreymer
    ikreymer committed Mar 11, 2015
    8f54132

  8. Merge branch 'master' of https://github.com/ikreymer/webarchive-indexing

    @ikreymer
    ikreymer committed Mar 11, 2015
    0378178

  9. tweak README

    @ikreymer
    ikreymer committed Mar 11, 2015
    fcfc82a

  10. Update README after refactor

    @ikreymer
    ikreymer committed Mar 11, 2015
    48fcf9f

  11. add .gitignore and LICENSE

    @ikreymer
    ikreymer committed Mar 11, 2015
    5c553b7

  12. more cleanup of run scripts, integrate seq upload into dosample job

    @ikreymer
    ikreymer committed Mar 11, 2015
    71ccd91
Commits on Mar 10, 2015

  1. refactoring: use cmdline options instead of fixed constants!

    @ikreymer
    ikreymer committed Mar 10, 2015 
    rename job files to end in job
add integrated samplecdx script for running samplecdxjob and converting to sequencefile
add seqfileutils
    57cb430
Older
You can’t perform that action at this time.