Question: How to recursively archive directories containing many PDF download links

[JKL213]

Title as is. I want to archive an Apache2 file server page (download.eversberg.eu) that has a ton of PDF files. Any convenient way to achieve this? I tried depth=1 and it does not really work. The PDF links still redirect to the original page.

[pirate]

ArchiveBox doesn't go more than 1 hop deep when using depth=1, so unless the links to the PDFs are directly on the page you're passing it, it's not going to recursively keep following links beyond the first hop.

You can do multiple passes if you need it to go deeper into a folder structure:

archivebox add --depth=1 'https://download.eversberg.eu'
archivebox list --csv=url --filter-type=domain download.eversberg.eu | archivebox add --depth=1
archivebox list --csv=url --filter-type=domain download.eversberg.eu | archivebox add --depth=1
archivebox list --csv=url --filter-type=domain download.eversberg.eu | archivebox add --depth=1
...
# repeat for however many levels deep you want to go

I recommend SiteSucker or wget's recursive feature, they may be better suited to this use case.

See this issue for more info: #191

[JKL213]

I'm sorry, my request was terribly worded. I'm looking to archive the site "download.eversberg.eu" which is essentially just a file host running on Apache2. I managed to solve my issue by just running individual archive tasks for every folder and using depth setting 1. I'm essentially just using for a convenient way to archive below depth 1.

[pirate]

No worries, I figured out what you meant after I visited the domain, edited my comment above 👍