Different results from ament_flake8 when run locally

[ct2034]

Description

I am not sure if this is the same problem as #318, but it might be.
I also get different results when running ament_flake8 locally vs in the github action (https://github.com/ros/diagnostics/actions/runs/3875359248/jobs/6607784527)

Expected Behavior

Github action and local should result in the same output.

Actual Behavior

Locally I get:

14 files checked
No problems found
...

To Reproduce

Locally I have also ros-rolling-ament-flake8 installed.
And run it with ament_flake8 $(colcon list --packages-select diagnostic_aggregator diagnostic_common_diagnostics diagnostic_updater self_test -p)

System (please complete the following information)

• OS: 20.04.1
• ROS 2 Distro: Rolling
• https://github.com/ct2034/diagnostics/tree/add/linting_diagnostic_common_diagnostics

Additional context

See ros/diagnostics#268

[christophebedard]

This might be related to ros-tooling/setup-ros#535.

setup-ros installs a lot of dependencies from pip, which tends to have newer versions compared to apt. However, as part of a typical ROS 2 installation, you install these dependencies from apt.

[ct2034]

I did think the same thing. I tried to manually update all packages in my environment that seemed relevant but could still not replicate the errors that I get from the action. Could it maybe be related to a configuration that is defined for flake8? Is there some custom configuration defined for it or does it just check the defaults? I was not able to find a custom config.

[ct2034]

Okay, you were right. I just realized that flake8 also has plugins. Following the list on https://docs.ros.org/en/humble/Installation/Alternatives/Ubuntu-Development-Setup.html#install-development-tools-and-ros-tools, I installed:

pip install --upgrade flake8-blind-except flake8-builtins flake8-class-newline flake8-comprehensions flake8-deprecated flake8-import-order flake8-quotes flake8-docstrings

And am now able to completely replicate the findings of the action 😄

[christophebedard]

Yeah, that allows you to reproduce locally, but I still do plan on changing setup-ros to more closely match a typical installation you get when following the official docs (ros-tooling/setup-ros#535).

[ct2034]

Yes, ok. That sounds like a good idea

[Cakem1x]

This discussion helped me quite a bit, thanks! I have the same issue, so I think I post this here instead of opening another issue:

I get different results when running ament_flake8 locally vs in the GitHub action (ros-tooling/action-ros-ci, within an installation via ros-tooling/setup-ros).
What confuses me is that the ros-tooling version I use (ros-tooling/setup-ros@0.7.4) should already include the fix (released into 0.7.0.) from @christophebedard.

Any idea what I could be missing?

Details:
I run ament_flake8 locally (or colcon test --packages-select my_pkg) and I get no errors. On Ubuntu 22.04.
However, the GitHub action (see below) fails due to flake8 errors.

jobs:
  build_and_test:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: ros-tooling/setup-ros@0.7.4
        with:
          required-ros-distributions: humble
      - uses: ros-tooling/action-ros-ci@0.3.12
        with:
          target-ros2-distro: humble
          rosdep-check: true

What I tried so far

1. To reproduce, use some python pkg e.g. with code that includes double quotes. Anything that uses the official ROS2 test_copyright.py should work...
2. Locally, I use ament_flake8 that came via rosdep install. Running it yields no error.
3. CI fails due to flake8 failure (e.g. ./test/test_copyright.py:22:26: Q000 Double quotes found but single quotes preferred rc = main(argv=[".", "test"]), among other failures)
4. I can get the same behavior locally by installing a bunch of flake8 plugins via pip (pip install --upgrade flake8-blind-except flake8-builtins flake8-class-newline flake8-comprehensions flake8-deprecated flake8-import-order flake8-quotes flake8-docstrings).
5. Now my ament_flake8 also fails locally (yay?)

[christophebedard]

I think the issue is that some flake8 plugins (in this case python3-flake8-quotes) are installed in CI by setup-ros but are not installed if you follow the Humble Ubuntu (Debian packages) installation instructions: https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html. If you follow the Humble Ubuntu (source, i.e., development) instructions, then you manually install python3-flake8-quotes: https://docs.ros.org/en/humble/Installation/Alternatives/Ubuntu-Development-Setup.html#install-development-tools-and-ros-tools.

python3-flake8-quotes was added as a proper <exec_depend> for ament_flake8: ament/ament_lint#454. Therefore, it gets installed if you install ros-${ROS_DISTRO}-ament-flake8, but that's only for Jazzy and later:

1. If you search for Package: ros-humble-ament-flake8 here, you see that python3-flake8-quotes is not a dependency: http://packages.ros.org/ros2/ubuntu/dists/jammy/main/binary-amd64/Packages
2. If you search for Package: ros-jazzy-ament-flake8 here, you see that python3-flake8-quotes is a dependency: http://packages.ros.org/ros2/ubuntu/dists/noble/main/binary-amd64/Packages

So my advice is to make sure to do this part of the Ubuntu (source) installation instructions: https://docs.ros.org/en/humble/Installation/Alternatives/Ubuntu-Development-Setup.html#install-development-tools-and-ros-tools. And, in newer distros, this should not be an issue.

[Cakem1x]

Alright, thanks a lot for your explanation!