How to crawl only one page

[nk-yuta]

Hello, I am new to Heritirx.
I am new to Heritirx.
I am not familiar with Heritirx and its configuration.

I would like to have Heritirx crawl only one web page and the embedded content it contains.
How can I configure the settings to achieve this?

What I have tried so far:.

I set maxHops in TooManyHopsDecideRule to 1. → Did not work as desired.
I set maxDocumentsDownload in the crawlLimitEnforcer to 2. → Only one page could be crawled, but CSS etc. were not downloaded.

I used machine translation for this post.

[ldko]

Hi @nk-yuta ,
To crawl only one page and its embedded content try setting maxHops to 0, which should download your seed but not take any navigational links from that seed:

 <bean id="scopeMaxHops" class="org.archive.modules.deciderules.TooManyHopsDecideRule">
   <property name="maxHops" value="0" />
 </bean>

Then use TransclusionDecideRule to indicate you want to get embedded content. For example you could set:

  <bean class="org.archive.modules.deciderules.TransclusionDecideRule">
    <property name="maxTransHops" value="2" />
    <property name="maxSpeculativeHops" value="1" />
   </bean>

Those are the main settings for getting one page with embeds. I recommend keeping maxDocumentsDownload at the default 0. Otherwise you are telling Heritrix to only download 2 URIs, which means it will stop the crawl before getting all of the URIs for the embedded content.

[nk-yuta]

Thanks for the advice!
I will try crawling with that setting.

[nk-yuta]

Thanks to your advice, I can submit my graduation thesis!
My subject of thesis is creating new web archiving system, so without this thread, I could not complete graduation reseach...
Unfortunately, I cannot share my graduation thesis, But I added your name to the acknowledgments.
I'm really thankful to you!