How to create a new log/report for a single class

[cgr71ii]

Hi!

Based on #284, I've created a cost class which currently I've tested that works. Trying to debug, I've created a logger like I've seen in other classes:

private static final Logger logger = Logger.getLogger(PUCCostAssignmentPolicy.class.getName());

The problem I'm facing is that the logging content is being shared across all the crawling jobs in the path /path_to_build_heritrix/heritrix-3.4.0-SNAPSHOT/heritrix_out.log. I'd like to redirect the content to a specific file like ${launchId}/logs/cost.log or ${launchId}/reports/cost.report (by the way, what's the difference in Heritrix when we say log or report?), is this possible to achieve easily?

I've tried to dig into the code and see how the logs and reports are implemented, but it seems that the reports are intended for general statistics or purposes, so I don't know if what I'm trying to achieve fits there. I've seen that reports are registered in StatisticsTracker.java and logs in CrawlerLoggerModule.java (and related to Logs.java). Since my case is rather simple, I think I shouldn't modify all this code just in order to implement the logging facility for a single class (of course, it would be different if I'd want to implement a generic and global report/log for cost functions, but it's not the case).

Thank you!

PS: I've seen very interesting questions/discussions in the issue section. How about enabling and using the discussions section? I think it would be very interesting in order to, for example, avoid to "forget" old questions. There's a "Q&A" section there that fits perfectly (e.g. https://github.com/bitextor/bitextor/discussions/categories/q-a)

[ato]

PS: How about enabling and using the discussions section?

I asked on the IIPC Slack and a few people agreed with this suggestion and nobody dissented so I've enabled the discussions section and moved most of the existing questions here. (Just answering this part as I'm not familiar with the logging/reporting code.)

[cgr71ii]

Hi! I've finally got it!

Example of a new log (not report) working: cgr71ii@c9be5b3

Main different I've observed between log and report: logs are updated frequently, reports need to be opened to trigger the update unless that shouldReportAtEndOfCrawl is set to false. I guess that the report goal is to have a file which is totally rewritten, and since we're crawling, we don't want that in order to avoid to slow down the disk operations. Since logs are intended to append text, this is not such a critic problem (hypothesis).

Main steps in order to create a new log:

1. Add entry in engine/src/main/java/org/archive/crawler/util/Logs.java.
2. Modify engine/src/main/java/org/archive/crawler/reporting/CrawlerLoggerModule.java and add your logger just like all the others are.
3. Create your formatter in engine/src/main/java/org/archive/crawler/io/ (in my case, I just copied engine/src/main/java/org/archive/crawler/io/StatisticsLogFormatter.java since is a generic formatter and I think it might fit almost all use cases).
4. Modify your Java classes which you want to use this formatter. You'll need to create a CrawlerLoggerModule (the setter will need to use @Autowired) instance and define Logger logger = crawlerLoggerModule.getYourLoggerInstance() and then you just have your logger which will log in a separate file instead of logging in heritrix_out.log all mixed up with all the instances which are logging independently.

PS: thanks for the discussions section! I think it fits very well in this project.

[cgr71ii]

For reports, I think that a similar method might be applied in the file engine/src/main/java/org/archive/crawler/reporting/StatisticsTracker.java, but not sure.

[cgr71ii]

Another difference that I think I've found between report and log files: reports content seems to be copied to a new job when a checkpoint is used/loaded, but the same scenario doesn't seem to apply to log files (i.e. content is not copied to the new job). I think this is something important to consider if a lot of content is being written to these files.