Scripts to generate profiles of various Web archives
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
benchmark
cdx
json
.gitignore
LICENSE
README.md
benchmark_analyzer.py
cdx_extract_profiler.py
cdx_profiler.py
cdx_transformation_analyze.r
config.ini
configold.ini
extract_benchmark_analyzer.py
extract_profiling_benchmarker.py
extract_suburi_generator.py
extract_summarize.r
key_generator.py
keyword_sample_profiler.py
lanl_profiler.py
main.py
mainold.py
newmain.py
policy_benchmark_analyzer.py
policy_profiling_benchmarker.py
policy_summarize.r
prefix_suffix_profiler.py
profile.py
profile_merger.py
profiling_benchmarker.py
requirements.txt
suburi_generator.py
summarize.r
test_suburi_generator.py
uri_sample_profiler.py
urir_growth_analyze.r

README.md

Archive Profiler

Scripts to generate profiles of various Web archives that will be saved in Archive Profiles Repository.

Running Profiler Script

To setup and run the Profiler script, please follow these steps:

Clone the repository.

$ git clone git@github.com:oduwsdl/archive_profiler.git

Change working directory.

$ cd archive_profiler

Install dependencies from the requirement file (add sudo before pip command if necessary.)

$ pip install -r requirements.txt

Run the script on the shipped sample cdx files.

$ python ./main.py cdx/*.cdx

If the script finishes without errors, it should save the profiles in the profiles folder. Now please update the config.ini file to reflect your collection. Then try to run profiler against your own cdx file(s). This will generate profiles for your collection and will save them in the profiles directory (it will overwrite existing files with the same name).

$ python ./main.py path/to/cdx/files/*.cdx