[BUG] `--errors` report error while normal run performs fine

[yubiuser]

I run webchanges once a day vai cron job with --errors to ... check for errors. There is one particular job that regularly reports (always the same) error, but during the normal run nothing sticks out. I tried to run it with --errors from cli but it performs fine.
Is there any difference when a job is executed vs when it runs via --errors? I particular, does a normal job try to fetch more than once in case an connection error occurs?

Would it make sense to offer a "file" reporter? I could image to run --errors and log a verbose output to a file but still send a notification to a chosen reporter (e.g. with a hint to check the logs). Like webchanges -v --errors mail --errorfile error.log ....

---

The job is

name: "Xenia Downloads"
url: "https://www.xenia-espresso.de/downloads/"
headers:
    User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
filter:
    - element-by-class: media-body
    - html2text:
       body_width: 0
       ignore_links: true
       ignore_images: true
       single-line-break: true
    - execute: paste -s -d' \n'
    - execute: paste -s -d' \n'
    - execute: awk 'BEGIN { FS = "__" } ; { print $1 $3 }'
    - strip

Reported error is

Error "[Errno-3] Try again": Xenia Downloads...

Version info.

Software:
• webchanges: 3.26.0
• CPython: 3.12.7 ('main', 'Nov 12 2024 02:17:58') GCC 13.2.1 20240309
• SQLite: 3.45.3

Installed PyPi dependencies:
• cryptography: 43.0.3

[mborsetti]

Thank you so very much for the bug report.

I am noticing something similar. The logic should be the same, so it's puzzling. Will make time to track it down.

[mborsetti]

I think I found the issue and fixed it. Would you be so kind to try the beta version to see if it works for you too?

pip install https://github.com/mborsetti/webchanges/archive/unreleased.tar.gz

I've also added a --log-file command line argument to save a log file but haven't tested it yet (and realize that it won't work unless you also use the -v argument; that will be fixed before release).

[yubiuser]

Thanks.

I've created a special build for anyone to try this change at ghcr.io/yubiuser/webchanges:pr-105

It will need a few days before I can say if it's fixed, my errors job runs only once a day.

---

P.S. Why don't you use github's pull request feature for new features, fixes, releases? This would make it easier for others to track changes and comment/review your commits.

[yubiuser]

I got the error again. The log output does not contain any useful information I think.

2024-11-26 08:00:01,579 cli[140300173589288] INFO: webchanges: 3.27.0b2 Copyright 2020- Mike Borsetti
2024-11-26 08:00:01,579 cli[140300173589288] INFO: CPython: 3.12.7 ('main', 'Nov 12 2024 02:17:58') GCC 13.2.1 20240309
2024-11-26 08:00:01,682 cli[140300173589288] INFO: System: Linux-6.1.0-27-amd64-x86_64-with
2024-11-26 08:00:02,442 storage[140300173589288] INFO: Loaded configuration from config.yaml
2024-11-26 08:00:02,443 storage[140300173589288] INFO: Using sqlite3 3.45.3 database at snapshots.db
2024-11-26 08:00:02,482 storage[140300173589288] INFO: Loaded 21 jobs from [PosixPath('jobs.yaml')]
2024-11-26 08:00:11,547 handler[140300097731384] INFO: Job 8: https://www.xenia-espresso.de/downloads/ started processing (UrlJob)
2024-11-26 08:00:11,551 jobs[140300097731384] INFO: Job 8: Sending GET request to https://www.xenia-espresso.de/downloads/
2024-11-26 08:00:11,551 jobs[140300097731384] INFO: Job 8: Using the HTTPX HTTP client library with HTTP/2 support
2024-11-26 08:00:16,663 handler[140300097731384] INFO: Job 8: https://www.xenia-espresso.de/downloads/ ended processing
2024-11-26 08:00:17,360 reporters[140300173589288] INFO: Submitting with pushover (<class 'webchanges.reporters.PushoverReport'>)
2024-11-26 08:00:22,455 command[140300173589288] INFO: Exiting with exit code 0

But I think it missed some information. For all other jobs I get a line with a http status code like

INFO: HTTP Request: GET https://XXXXXXXXXXXXX "HTTP/1.1 200 OK"

which is missing for the failing job 8.

[yubiuser]

This is how it looks like on days it's working

2024-11-25 08:00:02,979 handler[139719930932024] INFO: Job 8: https://www.xenia-espresso.de/downloads/ started processing (UrlJob)
2024-11-25 08:00:02,985 jobs[139719930932024] INFO: Job 8: Sending GET request to https://www.xenia-espresso.de/downloads/
2024-11-25 08:00:02,985 jobs[139719930932024] INFO: Job 8: Using the HTTPX HTTP client library with HTTP/2 support
2024-11-25 08:00:08,719 _client[139719930932024] INFO: HTTP Request: GET https://www.xenia-espresso.de/downloads/ "HTTP/2 200 OK"
2024-11-25 08:00:08,785 filters[139719930932024] INFO: Job 8: Applying filter element-by-class, subfilter(s) {'class': 'media-body'}
2024-11-25 08:00:08,856 filters[139719930932024] INFO: Job 8: Applying filter html2text, subfilter(s) {'body_width': 0, 'ignore_links': True, 'ignore_images': True, 'single-line-break': True}
2024-11-25 08:00:08,879 filters[139719930932024] INFO: Job 8: Applying filter execute, subfilter(s) {'command': "paste -s -d' \\n'"}
2024-11-25 08:00:08,881 filters[139719930932024] INFO: Job 8: Applying filter execute, subfilter(s) {'command': "paste -s -d' \\n'"}
2024-11-25 08:00:08,883 filters[139719930932024] INFO: Job 8: Applying filter execute, subfilter(s) {'command': 'awk \'BEGIN { FS = "__" } ; { print $1 $3 }\''}
2024-11-25 08:00:08,886 filters[139719930932024] INFO: Job 8: Applying filter strip, subfilter(s) {}
2024-11-25 08:00:08,886 handler[139719930932024] INFO: Job 8: https://www.xenia-espresso.de/downloads/ ended processing

[mborsetti]

Thanks for the logs. From both of them, it looks like the issue is the HTTP client receiving an error from the OS. Testing for errors is faster than when the data has to be saved, so that may be why you're only getting this error when using --error.

From an LLM:

This error message usually means there's a temporary problem with your network connection. Here's a breakdown and how to troubleshoot it:

What the error means:

Errno -3: This is a general network error code. It often translates to "Try again" or "Temporary failure in name resolution".
Name resolution: This refers to the process of converting a domain name (like "[invalid URL removed]") into an IP address that computers use to communicate.
Possible causes:

DNS problems: Your device might be having trouble contacting the Domain Name System (DNS) server that translates domain names to IP addresses.
Network connectivity issues: There could be a temporary interruption in your internet connection or a problem with your Wi-Fi or network settings.
Firewall or antivirus software: Security software might be interfering with your network connection.
Overloaded server: The server you're trying to reach might be temporarily unavailable due to high traffic.
Troubleshooting steps:

Check your internet connection: Make sure you have a stable internet connection. Try browsing to other websites or running a speed test.
Restart your network devices: Restart your modem, router, and the device you're using.
Try a different DNS server: You can temporarily change your DNS settings to use a public DNS server like Google Public DNS (8.8.8.8 and 8.8.4.4) or Cloudflare DNS (1.1.1.1 and 1.0.0.1).
Temporarily disable firewall/antivirus: See if the error goes away when you temporarily disable your firewall or antivirus software. If it does, you may need to adjust your security software settings.
Try again later: Sometimes the problem resolves itself. Wait a while and try again.

What operating system are you using? (e.g., Windows, macOS, Linux)

[yubiuser]

What operating system are you using? (e.g., Windows, macOS, Linux)

I'm using Linux, but run webchanges in a docker container. I changed the DNS server used by the container by explicitly setting the dns key in the docker-compose file. And it seems to be working better now. I'll remove the key in a few days and will observe if the errors return.

[yubiuser]

Ok, it's the DNS. Setting a DNS server via dns: in the docker compose file fixed the errors. After removal they re-appear.
However, it's still strange that it only happens on the --errors runs, and not during normal operation. Or is is only suppressed/silenced when --errors is absent?

[mborsetti]

My hypothesis is that --errors runs faster as it doesn't have to do any database reading and writing nor any reporting, so the DNS receives more hits per second than under a normal run.

Thanks for tracking this down!