Clang-tidy detects issues in code, but `WarningsAsErrors` pervents cpp-linter from reporting them

[FelipeGdM]

What events trigger your workflow?

name: cpp-linter-all
on:
  pull_request:
  push:

permissions:
  pull-requests: write
  contents: write

jobs:
  cpp-linter:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v5

      - name: Install dependencies
        run: sudo apt-get update && sudo apt-get install -y libc6 wget lsb-release build-essential software-properties-common gnupg cmake qtbase5-dev qt5-qmake libqt5opengl5-dev libprotobuf-dev mesa-common-dev libglu1-mesa-dev protobuf-compiler

      - name: Generate build files
        run: mkdir build && cd build && cmake .. && make proto_msgs_autogen VSSReferee_autogen
        # ... optionally setup build env to create a compilation database

      - uses: cpp-linter/cpp-linter-action@v2
        id: linter
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          ignore: '.github|build|src|!src/world/entities/vision'
          style: ''  # disable clang-format checks.
          tidy-checks: '' # Use .clang-tidy config file.
          step-summary: true
          thread-comments: false
          no-lgtm: false
          files-changed-only: false
          database: 'build/'
          verbosity: 'debug'

      - name: Debug output variables
        run: |
          echo "Checks failed: ${{ steps.linter.outputs.checks-failed }}"
          echo "Tidy checks failed: ${{ steps.linter.outputs.clang-tidy-checks-failed }}"
          echo "Files analyzed: ${{ steps.linter.outputs }}"

      - name: Fail fast?!
        if: steps.linter.outputs.clang-tidy-checks-failed > 0
        run: exit 1

What OS does your workflow use?

runs-on: ubuntu-22.04

How is cpp-linter-action configured?

- uses: cpp-linter/cpp-linter-action@v2
        id: linter
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          ignore: '.github|build|src|!src/world/entities/vision'
          style: ''  # disable clang-format checks.
          tidy-checks: '' # Use .clang-tidy config file.
          step-summary: true
          thread-comments: false
          no-lgtm: false
          files-changed-only: false
          database: 'build/'
          verbosity: 'debug'

What was the unexpected behavior?

A lot of errors are detected by clang-tidy, but the output message is "No problems need attention."

The action fails to count the total amount of errors

Final lines of log

DEBUG:CPP Linter:clang-tidy made the following summary:
13115 warnings generated.
Suppressed 13112 warnings (13112 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
3 warnings treated as errors

Value of steps.linter.outputs.checks-failed and steps.linter.outputs.clang-tidy-checks-failed

Checks failed: 0
Tidy checks failed: 0

The complete run is available at https://github.com/futebol-mini/VSSReferee/actions/runs/18019987310

[2bndy5]

The logs don't even get to the "posting results" step. I think there might be a stack overflow happening.

Most of the actual errors we run into is because people throw a ridiculously large number of files (with known erroneous syntax) at cpp-linter. I'm this case you've managed to throw a relatively small number of files riddled with a lot of errors in each.

Suggestions

1. Limit the number of files you analyze. We do offer lines-changed-only and ignore options.
2. Start addressing clang-tidy errors instead of ignoring them. I can understand if this is being phased in, but that doesn't excuse the "scan everything all at once" config.
3. Adjust your .clang-tidy config to reflect your expectations. If some warnings/errors are acceptable, then it would make sense to configure clang-tidy not to report them.

If you're hoping for some changes on our end to compensate for a ton of errors from a ton of files, then that might be an indefinite wait.

[2bndy5]

The logs don't even get to the "posting results" step.

Upon second look, the step summary is posted. So, cpp-linter is getting to that step, but the CI logs don't show it happening. This means it is hard to diagnose what exactly is going on. I'll have to try and reproduce locally just to see the entire logs.

[FelipeGdM]

Hi @2bndy5! Thank you for your reply!

I have just assumed the responsibility on maintaining this project and tried to put it on tracks as no standards were followed before. As you could see, there are a lot of problems in the codebase and they are not going to be fixed overnight.

I think that enabling cpp-linter for modified files and forcing a green check in every PR is a smooth way of packaging fixes as files are modified with new features

If you're hoping for some changes on our end to compensate for a ton of errors from a ton of files, then that might be an indefinite wait.

I'm sorry if I sounded rude in my first comment, I should have contextualize better the context of the project

I tried to troubleshoot the problem, so I reduced the scope of the scan to a small set of files and even tried different versions of cpp-lint. None of that solved the issue, so I suspected that a deeper problem may be present

[2bndy5]

I still suspect that cpp-linter is overflowing memory available to it, but I think that would cause an exit 1 situation. Even if we save all the aggregated info to temp cache files, the problem would still exist when the cache files are loaded to summarize the clang-tidy outputs.

FWIW, all comments posted on GH have a length limit (65535 bytes). The step-summary feature may not have the same limit (and no extra permissions needed).

I'm sorry if I sounded rude in my first comment, I should have contextualize better the context of the project

No offense was taken nor intended. I have a (bad) habit of being brutally honest.

I have just assumed the responsibility on maintaining this project and tried to put it on tracks as no standards were followed before.

This does help to know. And I sympathize with your efforts.

Have you tried running clang-tidy's auto-fix feature?

clang-tidy -h

 --fix                          -
                                   Apply suggested fixes. Without -fix-errors
                                   clang-tidy will bail out if any compilation
                                   errors were found.
  --fix-errors                   -
                                   Apply suggested fixes even if compilation
                                   errors were found. If compiler errors have
                                   attached fix-its, clang-tidy will apply them as
                                   well.
  --fix-notes                    -
                                   If a warning has no fix, but a single fix can
                                   be found through an associated diagnostic note,
                                   apply the fix.
                                   Specifying this flag will implicitly enable the
                                   '--fix' flag.

Note

clang-tidy fixes conform to a clang-format style (LLVM by default).

  --format-style=<string>        -
                                   Style for formatting code around applied fixes:
                                     - 'none' (default) turns off formatting
                                     - 'file' (literally 'file', not a placeholder)
                                       uses .clang-format file in the closest parent
                                       directory
                                     - '{ <json> }' specifies options inline, e.g.
                                       -format-style='{BasedOnStyle: llvm, IndentWidth: 8}'
                                     - 'llvm', 'google', 'webkit', 'mozilla'
                                   See clang-format documentation for the up-to-date
                                   information about formatting styles and options.
                                   This option overrides the 'FormatStyle` option in
                                   .clang-tidy file, if any.

[FelipeGdM]

Hi @2bndy5!

Have you tried running clang-tidy's auto-fix feature?

I applied all possible auto fixes, but it wasn't enough to solve the issue. I also tried to narrow down the scope of the analysis to just one file and it also didn't solve the problem

Upon further investigation, I found out that disabling WarningsAsErrors inside .clang-tidy was the solution. Is this behavior expected? If it is, we may close this issue

https://github.com/futebol-mini/VSSReferee/actions/runs/18073729711

Thank you very much for your support!

[2bndy5]

Upon further investigation, I found out that disabling WarningsAsErrors inside .clang-tidy was the solution. Is this behavior expected?

Interesting observation!
Our python code that runs clang-tidy per file expects the all lint violations to be printed to stdout. Meaning only clang-tidy output via stdout is reported. If clang-tidy only prints everything to stderr, then it is just logged and not parsed.

This problem implies a solution that complicates our parsing strategy because the logs about suppressed violations (and missing a compilation database) is always printed to stderr regardless.

I need to investigate this further. I suspect that LLVM changed where the log output is (conditionally) piped in a particular version of clang... I see you are using clang v18, so I can start there and work my backwards to v10.

[2bndy5]

I suppose as a workaround we can append to the clang-tidy command invocation:

clang-tidy <other-args ...>  --warnings-as-errors=-* <filename>

But I'm not sure that will work as expected (or consistently).

[2bndy5]

There might be something else afoot here. I quickly used nushell to see where the errors were printed:

 (^uvx clang-tidy@18.1.8 --warnings-as-errors=* demo.cpp) | complete
╭────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ stdout     │ C:\dev\cpp-linter\tests\demo\demo.cpp:2:1: error: included header demo.hpp is not used directly               │
│            │ [misc-include-cleaner,-warnings-as-errors]                                                                    │
│            │     2 | #include "demo.hpp"                                                                                   │
│            │       | ^~~~~~~~~~~~~~~~~~~                                                                                   │
│            │     3 |                                                                                                       │
│            │ #include <stdio.h>                                                                                            │
│            │ C:\dev\cpp-linter\tests\demo\demo.cpp:10:13: error: statement should be inside braces                         │
│            │ [readability-braces-around-statements,-warnings-as-errors]                                                    │
│            │    10 |     for (;;) break;                                                                                   │
│            │       |             ^                                                                                         │
│            │                                                                                                               │
│            │       |              {                                                                                        │
│            │                                                                                                               │
│ stderr     │ 2488 warnings generated.                                                                                      │
│            │ Suppressed 2486 warnings (2482 in non-user code, 4 with check filters).                                       │
│            │ Use                                                                                                           │
│            │ -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from   │
│            │ system headers as well.                                                                                       │
│            │ 2 warnings treated as errors                                                                                  │
│            │                                                                                                               │
│ exit_code  │ 1                                                                                                             │
╰────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

The lint violations still get printed to stdout using clang-tidy v18.1.8. I'm not sure why this WarngsAsErrors might be preventing accurate reports from the action.

[2bndy5]

[misc-include-cleaner,-warnings-as-errors]

Note, our parsing also does not support this format. It expects only 1 rule in square brackets. The ,-warnings-as-errors bit breaks the behavior that automatically hyperlinks to clang's explanation pages.

I think this is the part that breaks our parsing. Our current regex pattern does not account for a , in the square brackets. This should be a rather easy fix 😌