The `Vale` action fails in forks despite `fail_on_error: false`

[paddyroddy]

Describe the Bug

I spotted this in #545. It seems that this is a known limitation of reviewdog which is used by Vale. The reviewdog README explains the problem, see issues errata-ai/vale-action#89, reviewdog/reviewdog#411.

Essentially the Vale action switches to annotations from a fork to avoid the security issues which GitHub has a very low limit of 10 allowed https://github.com/orgs/community/discussions/68471. Currently, on main we have 16 total warnings/errors, which is >10. So that means in the diff it can only display 10 and then errors. So the failure we're seeing is not a Vale error, nor a reviewdog error, but a GitHub Action known limitation failure.

docs/pages/ci.md
 30:94  warning  'Very' is a weasel word!  write-good.Weasel
 30:94  error    Remove 'Very'.            proselint.Very


 CONTRIBUTING.md
 41:4  warning  'quite' is a weasel word!  write-good.Weasel


 docs/pages/community.md
 8:1  error  Don't start a sentence with     write-good.ThereIs
             'There are'.


 docs/pages/docs.md
 8:49    warning  'very' is a weasel word!  write-good.Weasel
 8:49    error    Remove 'very'.            proselint.Very
 31:116  warning  'very' is a weasel word!  write-good.Weasel
 31:116  error    Remove 'very'.            proselint.Very


 docs/pages/libraries/jupyter-notebooks.md
 29:1  error  Don't start a sentence with     write-good.ThereIs
              'There is'.


 docs/pages/libraries/parallel-async.md
 29:119  error  Consider using the '…' symbol   proselint.Typography
                instead of '...'.


 docs/pages/virtual.md
 16:104  warning  'very' is a weasel word!  write-good.Weasel
 16:104  error    Remove 'very'.            proselint.Very


 docs/pages/packaging.md
 10:1  warning  'several' is a weasel word!  write-good.Weasel


 tutorial.md
 10:34    error    Consider using the '…' symbol   proselint.Typography
                   instead of '...'.
 201:1    error    Don't start a sentence with     write-good.ThereIs
                   'There are'.
 201:269  warning  'extremely' is a weasel word!   write-good.Weasel

✖ 9 errors, 7 warnings and 0 suggestions in 166 files.

As a temporary workaround one could change the filter_mode to something other than nofilter so that we get under the 10. This would still mean when a new PR introduces more than 10 failures it would hit this same error, but that might not be a bad thing. It would encourage us to change our rules (i.e. turn rules on/off), and actually stick to them.

To Reproduce

No response

Expected Behaviour

No response

Actual Behaviour

No response

Version In Use

v1.0.1

Additional Context

- Cookiecutter version:
- Operating system:
- Python version:

[paddyroddy]

Have discussed and investigated this with @samcunliffe. We've decided this is something we can't fix/won't fix, and perhaps is not an issue.

What happens when an internal contributor makes a failing vale change?
The vale action won't fail in the event that one of us makes a known "failing" change. It will give a neutral status check. However, it will still present us with annotations so we know what to change (which is good).

What happens when an external contributor makes a failing vale change?
The vale action will fail in the event that someone else makes a known "failing" change. It will give a failing status check. It will also present us with annotations so we know what to change (which is good).

...

In any case, if the PR is merged into main these changes are now the source and hence through diff_check (see #547) would no longer be presented as annotations.