Heritrix 3.4.0-SNAPSHOT-2022-03-08T19:15:59Z keeps pausing.. #587

cslr · 2023-10-16T17:37:53Z

cslr
Oct 16, 2023

Heritrx keeps pausing during crawls for no good reason. It may crawl something like 1GB and then pause again and again.

Only error message I see is "WARNING politessDelay unset" but I don't find documentation how to set politessDelay.

Answered by anjackson

The politenessDelay warning is irritating but should not be the cause of the pause.

As far as I know, pausing can happen for a couple of reasons:

You have a DiskSpaceMonitor configured (e.g.) and this is pausing the crawl because you don't have enough disk space given how that bean is configured.
You have less than 5GB of free space, and you are using the default value of the FREE_DISK parameter for the underlying JE-BDB. See #340 for some information on that, but I personally would not recommend running Heritrix without plenty of space disk space.

If neither of these is consistent with what you're seeing then it sounds like a bug. In which case, some more detailed logs might help.

anjackson · 2023-10-25T22:36:14Z

The politenessDelay warning is irritating but should not be the cause of the pause.

As far as I know, pausing can happen for a couple of reasons:

You have a DiskSpaceMonitor configured (e.g.) and this is pausing the crawl because you don't have enough disk space given how that bean is configured.
You have less than 5GB of free space, and you are using the default value of the FREE_DISK parameter for the underlying JE-BDB. See Disk usage is not within je.maxDisk or je.freeDisk limits and write operations are prohibited #340 for some information on that, but I personally would not recommend running Heritrix without plenty of space disk space.

If neither of these is consistent with what you're seeing then it sounds like a bug. In which case, some more detailed logs might help.

0 replies