[FEATURE] Support custom reporters

[knutwannheden]

Is your feature request related to a problem? Please describe.
I've been using urlwatch up until now and just discovered this fork. Very nice! One crucial feature that is missing however is support for custom reporters. urlwatch supports this through the --hooks option. While it appears I can also with webchanges specify this option, it later tells me that the corresponding configuration in the file referenced by --config is invalid.

Here is the error that was reported:

ValueError: Unrecognized directive(s) in the configuration file urlwatch-errors.yaml:
report:
  custom_slack:
    channel: C02PC0NG111
    enabled: false
    token: xoxb-aaaaaaaaaaa-bbbbbbbbbbbbb-ffffffffffffffffffffffff

Describe the solution you'd like
I would like to be able to configure my custom reporters using the YAML config file as with urlwatch.

Describe alternatives you've considered
If webchanges supports some equivalent alternative that I missed, then that of course also works for me.

Additional context
And for completeness sake here is my hooks.py file:

from datetime import datetime

from slack_sdk import WebClient
from urlwatch import reporters


class CustomSlackReporter(reporters.TextReporter):
    """Custom Slack reporter that uploads diff as file to Slack"""

    __kind__ = 'custom_slack'

    def submit(self):
        text = '\n'.join(super().submit())
        if text:
            slack_token = self.config['token']
            client = WebClient(token=slack_token)
            timestamp = datetime.now().strftime("%d.%m.%Y %H:%M:%S")
            response = client.files_upload(
                channels=self.config['channel'],
                filetype="diff",
                content=text,
                initial_comment="Report " + timestamp,
                title=f"report-{datetime.now().strftime('%d.%m.%Y')}.txt"
            )

[mborsetti]

Hello Knut,

Thanks. I think you found a bug -- I will take a deeper look into it, but it would appear that the checking for typos in the config file is made before the new classes in hooks.py are loaded. I apologize for the unintended issue and will try to publish a fix soon.

[knutwannheden]

I think I have also found another bug related to encoding when using the shellpipe filter. I will investigate a bit further and report another issue.

(webchanges really offers some very nice features on top of urlwatch. I was for a while considering creating my own fork, but it looks like that won't be necessary now 😄 . Possibly I can contribute some ideas or maybe even some code here.)

[mborsetti]

Fixed in Release 3.9.1

[mborsetti]

I think I have also found another bug related to encoding when using the shellpipe filter. I will investigate a bit further and report another issue.

(webchanges really offers some very nice features on top of urlwatch. I was for a while considering creating my own fork, but it looks like that won't be necessary now 😄 . Possibly I can contribute some ideas or maybe even some code here.)

Thanks very much!! Would love any contributions!!!!!

[knutwannheden]

Hmmm... I just tested the new version, but I still get the same error. My config file looks like this:

display:
  error: false
  new: false
  unchanged: false
report:
  custom_slack:
    enabled: true
    channel: C02PC0NG111
    token: xoxb-aaaaaaaaaaa-bbbbbbbbbbbbb-ffffffffffffffffffffffff

I still get an error like this:

  File "C:\Users\knutw\AppData\Roaming\Python\Python310\Scripts\webchanges.exe\__main__.py", line 7, in <module>
  File "C:\Users\knutw\AppData\Roaming\Python\Python310\site-packages\webchanges\cli.py", line 232, in main
    urlwatch_command.run()
  File "C:\Users\knutw\AppData\Roaming\Python\Python310\site-packages\webchanges\command.py", line 675, in run
    self.urlwatcher.config_storage.load()
  File "C:\Users\knutw\AppData\Roaming\Python\Python310\site-packages\webchanges\storage.py", line 689, in load
    self.check_for_unrecognized_keys(config)
  File "C:\Users\knutw\AppData\Roaming\Python\Python310\site-packages\webchanges\storage.py", line 676, in check_for_unrecognized_keys
    raise ValueError(
ValueError: Unrecognized directive(s) in the configuration file urlwatch-errors.yaml:
report:
  custom_slack:
    enabled: true
    channel: C02PC0NG111
    token: xoxb-aaaaaaaaaaa-bbbbbbbbbbbbb-ffffffffffffffffffffffff
Check for typos (documentation at https://webchanges.readthedocs.io/)

[knutwannheden]

I also wanted to test the new Playwright support (I never got Pyppeteer working), but also that I couldn't get to work. Did possibly something go wrong with the release process?

[knutwannheden]

I also wanted to test the new Playwright support (I never got Pyppeteer working), but also that I couldn't get to work. Did possibly something go wrong with the release process?

I suppose not. webchanges --version in any case reports 3.9.1.

[mborsetti]

It's not working because you have the wrong import in your hooks.py file, so it doesn't recognize that class as being a valid one:

from urlwatch import reporters

should read

from webchanges import reporters

I will add this to the migration documentation. Complex use cases are not as straightforward of a migration, thanks for sharing your experience.

[mborsetti]

I also wanted to test the new Playwright support (I never got Pyppeteer working), but also that I couldn't get to work. Did possibly something go wrong with the release process?

Working fine here (in production), both Windows and Ubuntu. What errors are you getting?

[knutwannheden]

from urlwatch import reporters

🤦‍♂️ I should have noticed that.

Working fine here (in production), both Windows and Ubuntu. What errors are you getting?

Every time I've tried this it appears to download Chrome and start it. After that it just hangs indefinitely.

[mborsetti]

Every time I've tried this it appears to download Chrome and start it. After that it just hangs indefinitely.

Have you tried to set the job sub-directive headless: false (in addition of running with --verbose) to see what's happening with Playwright/Chrome? It's documented here until Playwright goes into production.