Failed to start bean 'bdb'; nested exception is java.lang.IllegalStateException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11)

[albertkao9]

I like to download the webpage https://netbasal.com/using-the-angular-http-client-in-angular-v15-f4bec3c11926
for offline viewing.

I modify jobs/jobnetbasalhttpclient/crawler-beans.cxml as follows:

metadata.operatorContactUrl=https://github.com/albertkao9/angularnetbasal

# URLS HERE
https://netbasal.com/using-the-angular-http-client-in-angular-v15-f4bec3c11926

Running jobnetbasalhttpclient get:

2023-02-17T14:51:09.986Z SEVERE Failed to start bean 'bdb'; nested exception is java.lang.IllegalStateException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11) /home/ak/sw/heritrix-3.4.0-20220727/./jobs/jobnetbasalhttpclient/state Can not register /home/ak/sw/heritrix-3.4.0-20220727/./jobs/jobnetbasalhttpclient/state or its sub-directories to WatchService. UNEXPECTED_EXCEPTION: Unexpected internal Exception, may have side effects. Environment is invalid and must be closed. (in thread 'jobnetbasalhttpclient launchthread')
2023-02-17T14:51:09.986Z SEVERE Failed to start bean 'bdb'; nested exception is java.lang.IllegalStateException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11) /home/ak/sw/heritrix-3.4.0-20220727/./jobs/jobnetbasalhttpclient/state Can not register /home/ak/sw/heritrix-3.4.0-20220727/./jobs/jobnetbasalhttpclient/state or its sub-directories to WatchService. UNEXPECTED_EXCEPTION: Unexpected internal Exception, may have side effects. Environment is invalid and must be closed.
2023-02-17T14:51:09.985Z INFO Job instance discarded
2023-02-17T14:51:09.973Z INFO Job launched
2023-02-17T14:51:02.510Z INFO Job instantiated

ls -l jobs/jobnetbasalhttpclient/20230217145109/logs/

total 0
-rw-rw-r-- 1 ak ak 0 Feb 17 09:51 alerts.log
-rw-rw-r-- 1 ak ak 0 Feb 17 09:51 crawl.log
-rw-rw-r-- 1 ak ak 0 Feb 17 09:51 nonfatal-errors.log
-rw-rw-r-- 1 ak ak 0 Feb 17 09:51 progress-statistics.log
-rw-rw-r-- 1 ak ak 0 Feb 17 09:51 runtime-errors.log
-rw-rw-r-- 1 ak ak 0 Feb 17 09:51 uri-errors.log

heritrix version 3.4.0-20220727

[Mareike48]

I've had the same issue - solved the problem by using an older Heritrix version. Works without any issues, maybe that helps.

[anjackson]

The Environment is invalid and must be closed. error means Heritrix can't create/use the on-disk databases it uses to track the crawl. This could happen because:

• The crawl is configured to use a directory path that is invalid, e.g. wrong permissions.
• The crawl is trying to re-use the state directory of a significantly older version of Heritrix, which is not compatible.
• The crawl is trying to re-use a state directory that has been corrupted. This can happen due to system failures or bugs in Heritrix itself.

Heritrix should always start up fine with a new, empty state folder with appropriate file permissions. We often find after a year or so we need to delete the state folders and start afresh.

[yashvora93]

I am facing the same issue while using the latest docker image available here. Does this have something to do with the Operating System. I am on Apple M1 Pro chip. For the permissions, I have given all the necessary permissions (777) to the jobs folder which is mounted