[BUG] `--error` reports no data while there is data

[yubiuser]

/data/webchanges # su -c 'webchanges --error --jobs jobs.yaml --config config.yaml --database snapshots.db'
Jobs with errors or returning no data (after unmodified filters, if any)
   in jobs file jobs.yaml:
 13: No data: Pushover Blog (https://blog.pushover.net/)
 16: No data: zerolog github tags (https://api.github.com/repos/rs/zerolog/tags)
  2: No data: Matrica (http://matrica.de/service/service_aktuell.html)
--
Checked 26 enabled jobs for errors in 4.9 seconds.

/data/webchanges # su -c 'webchanges --test 13 --jobs jobs.yaml --config config.yaml --database snapshots.db'
===========================================================================
01. NEW: Pushover Blog
===========================================================================

---------------------------------------------------------------------------
NEW: Pushover Blog (https://blog.pushover.net/)
---------------------------------------------------------------------------

  *  Setup
  *  Configuration and Additional E-Mail Addresses
  *  Pushover for Teams
  *  Inactivity Monitors
  *  Creating a Monitor
  *  Alert Options
  *  Timeline
  *  Why is SMS Backup Being Discontinued?
  *  Push Notification Enhancements
  *  Support
  *  Pushover Notifications are Encrypted
  *  New Apps View
  *  Expiring Messages
  *  Support Development
---------------------------------------------------------------------------

Excerpt from webchanges -v --error

/data/webchanges # su -c 'webchanges -v --error --jobs jobs.yaml --config config.yaml --database snapshots.db' | grep 13
2025-05-13 09:10:16,490 cli[140356249979688] INFO: webchanges: 3.30.0 Copyright 2020- Mike Borsetti
2025-05-13 09:10:16,490 cli[140356249979688] INFO: CPython: 3.13.3 ('main', 'May  9 2025 23:41:23') GCC 14.2.0
2025-05-13 09:10:16,591 cli[140356249979688] INFO: System: Linux-6.1.0-28-amd64-x86_64-with
2025-05-13 09:10:17,258 cli[140356249979688] INFO: Hooks file /root/.config/webchanges/hooks.py does not exist or is not a file
2025-05-13 09:10:17,284 storage[140356249979688] INFO: Loaded configuration from config.yaml
2025-05-13 09:10:17,285 storage[140356249979688] INFO: Using sqlite3 3.48.0 database at snapshots.db
2025-05-13 09:10:17,292 jobs[140356249979688] INFO: Job 3: Replacing deprecated directive 'filter' with 'filters'
2025-05-13 09:10:17,343 storage[140356249979688] INFO: Loaded 26 jobs from jobs.yaml

2025-05-13 09:10:17,347 handler[140356179483448] INFO: Job 13: https://blog.pushover.net/ started processing (UrlJob)
 processing (UrlJob)
2025-05-13 09:10:17,350 jobs[140356179483448] INFO: Job 13: Sending GET request to https://blog.pushover.net/
2025-05-13 09:10:17,353 jobs[140356179483448] INFO: Job 13: Using the HTTPX HTTP client library with HTTP/2 support
2025-05-13 09:10:18,257 handler[140356179483448] INFO: Job 13: https://blog.pushover.net/ ended processing
2025-05-13 09:10:21,677 command[140356249979688] INFO: Exiting with exit code 0
 13: No data: <url headers={} method='GET' ignore_connection_errors=True ignore_http_error_codes='408, 429, 500, 502, 503, 504' filters=[{'element-by-class': 'update'}, {'element-by-tag': 'h2'}, {'html2text': {'body_width': 0, 'single-line-break': True, 'ignore_links': True, 'ignore_images': True}}, {'re.sub': {'pattern': '(?m)^##', 'repl': '  * '}}] index_number=13 name='Pushover Blog' url='https://blog.pushover.net/'

Job config

name: Pushover Blog
url: https://blog.pushover.net/
filters:
    - element-by-class: update
    - element-by-tag: h2
    - html2text:
        body_width: 0
        single-line-break: true 
        ignore_links: true
        ignore_images: true
    - re.sub:
        pattern: (?m)^##
        repl: '  * '

[mborsetti]

Sorry, this has been on my radar for way too long.

I am seeing the same issue, and I am hypothesizing that parallelism may be overwhelming responses (--error jobs run faster); I am going to try running --error with --max-workers 1 and see if that is the issue.

[yubiuser]

Good idea. I'll try the same and report back

[yubiuser]

Also with --max-workers 1 the errors occur

[mborsetti]

Hi @yubiuser,

OK, I think I have finally squashed the bug!

The legacy code has a convoluted way of handling websites that return a HTTP 304 "Not Modified" status code, which indicates that the requested resource has not been modified since the last time the client accessed it, by raising a custom Exception. My code for --error was not handling this correctly, and was simply reporting the error.

I will have the fix in the next release, which is practically ready to go; if you have any time, could you ride-test it?

pip install https://github.com/mborsetti/webchanges/archive/unreleased.tar.gz

The full release notes are at https://github.com/mborsetti/webchanges/blob/unreleased/RELEASE.rst

Thanks for your invaluable contributions!

[yubiuser]

Works fine. I pushed an 'unreleased' docker image to ghcr.io/yubiuser/webchanges:pr-143 (yubiuser/webchanges-docker#143)

and voilà

/data/webchanges # su -c 'webchanges --errors --jobs jobs.yaml --config config.yaml --database snapshots.db' web
changes
Jobs with errors or returning no data (after unmodified filters, if any)
   in jobs file jobs.yaml:
--
Checked 23 enabled jobs for errors in 23 seconds.

[mborsetti]

Awesome, thanks! v3.31.0 released!