IGNORE_GITIGNORED_FILES not applied for cfn-lint #2955

ashokm · 2023-09-14T07:50:47Z

Describe the bug
The documentation states that the IGNORE_GITIGNORED_FILES env var has a default setting of true and that:

If set to true, MegaLinter will skip files ignored by git using .gitignore file

This has been working as expected in v6, but has stopped working when v7 was released.
After v7, we see cfn-lint not behaving as expected and it is now including directories that are set in .gitignore.

To Reproduce
For a CDK-based project as an example, add cdk.out to .gitignore in the project repository.
Run MegaLinter v7 (running v7.3.0 at time of writing).
You'll see in the output (in Azure Pipelines in this case) that /tmp/lint/cdk.out/manifest.json, /tmp/lint/cdk.out/tree.json etc are listed as being .gitignored in this line:

- Excluding .gitignored files

However, after that, these files are still included in the linting scan:

...
2023-09-14T07:34:59.9630900Z ##[group]�[31m❌ Linted [CLOUDFORMATION] files with [cfn-lint]: Found 1 error(s) - (6.41s)�[0m (expand for details)
2023-09-14T07:34:59.9632415Z - Using [cfn-lint v0.79.6] https://megalinter.io/7.3.0/descriptors/cloudformation_cfn_lint
2023-09-14T07:34:59.9633131Z - MegaLinter key: [CLOUDFORMATION_CFN_LINT]
2023-09-14T07:34:59.9635473Z - Rules config: [.cfnlintrc.yml]
2023-09-14T07:34:59.9636044Z - Number of files analyzed: [4]
2023-09-14T07:34:59.9636477Z --Error detail:
2023-09-14T07:34:59.9636810Z E1001 Missing top level template section Resources
2023-09-14T07:34:59.9716728Z cdk.out/manifest.json:1:1
2023-09-14T07:34:59.9717319Z 
2023-09-14T07:34:59.9717740Z E1001 Top level template section version is not valid
2023-09-14T07:34:59.9717985Z cdk.out/manifest.json:2:3
2023-09-14T07:34:59.9718177Z 
2023-09-14T07:34:59.9718350Z E1001 Top level template section artifacts is not valid
2023-09-14T07:34:59.9718554Z cdk.out/manifest.json:3:3
...
2023-09-14T07:34:59.9721633Z E1001 Missing top level template section Resources
2023-09-14T07:34:59.9721814Z cdk.out/tree.json:1:1
2023-09-14T07:34:59.9721883Z 
2023-09-14T07:34:59.9722044Z E1001 Top level template section version is not valid
2023-09-14T07:34:59.9722212Z cdk.out/tree.json:2:3
2023-09-14T07:34:59.9722295Z 
2023-09-14T07:34:59.9722438Z E1001 Top level template section tree is not valid
2023-09-14T07:34:59.9722619Z cdk.out/tree.json:3:3
...

Additional context

No custom mega-linter.yml configuration file is being used
No custom .cfnlintrc.yml configuration file is being used

The text was updated successfully, but these errors were encountered:

nvuillam · 2023-09-18T22:59:20Z

@ashokm that's strange, cfn-lint is called with list_of_files so .gitignore files should be indeed ignored (https://megalinter.io/latest/descriptors/cloudformation_cfn_lint/#how-the-linting-is-performed )

Please could you run your workflow with LOG_LEVEL: DEBUG so we can investigate ?

ashokm · 2023-09-19T07:30:24Z

Any specific part of the output you're interested in?
There is a lot of information in the logs, some of which I cannot share for obvious reasons.
Below is an extract of some parts of the log with LOG_LEVEL set to 'DEBUG':

...
...
2023-09-19T06:37:37.7853574Z Listing all files in directory [/tmp/lint], then filter with:
2023-09-19T06:37:37.7853761Z Root dir content:
2023-09-19T06:37:37.7854126Z - .checkov.yml
2023-09-19T06:37:37.7854319Z - .gitignore
2023-09-19T06:37:37.7854505Z - .npmrc
2023-09-19T06:37:37.7854823Z - README.md
2023-09-19T06:37:37.7855127Z - azure-pipelines.yml
2023-09-19T06:37:37.7855330Z - cdk.context.json
2023-09-19T06:37:37.7855517Z - cdk.json
2023-09-19T06:37:37.7855705Z - jest.config.js
2023-09-19T06:37:37.7855887Z - package-lock.json
2023-09-19T06:37:37.7856079Z - package.json
2023-09-19T06:37:37.7856268Z - tsconfig.json
2023-09-19T06:37:37.7867150Z All found files before filtering:
2023-09-19T06:37:37.7867425Z - .checkov.yml
2023-09-19T06:37:37.7867643Z - .devcontainer/Dockerfile
2023-09-19T06:37:37.7867879Z - .devcontainer/devcontainer.json
2023-09-19T06:37:37.7868208Z - .gitignore
2023-09-19T06:37:37.7868370Z - .npmrc
2023-09-19T06:37:37.7868561Z - .vscode/settings.json
2023-09-19T06:37:37.7868750Z - README.md
2023-09-19T06:37:37.7868943Z - azure-pipelines.yml
2023-09-19T06:37:37.7869130Z - bin/blah-cdk-stack.d.ts
2023-09-19T06:37:37.7869330Z - bin/blah-cdk-stack.js
2023-09-19T06:37:37.7869527Z - bin/blah-cdk-stack.ts
2023-09-19T06:37:37.7869718Z - cdk.context.json
2023-09-19T06:37:37.7870107Z - cdk.json
2023-09-19T06:37:37.7874031Z - cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v2-handler.js
2023-09-19T06:37:37.7874489Z - cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v3-handler.bundled.js
2023-09-19T06:37:37.7874900Z - cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/index.js
2023-09-19T06:37:37.7875276Z - cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/shared.js
2023-09-19T06:37:37.7875686Z - cdk.out/asset.b992a5b1efd6715c28d74eda79615c6b4c3cf54e6364f2b2166dacf0f6c68602/update-blah-ip-set.py
2023-09-19T06:37:37.7876075Z - cdk.out/cdk.out
2023-09-19T06:37:37.7876288Z - cdk.out/manifest.json
2023-09-19T06:37:37.7876497Z - cdk.out/tree.json
2023-09-19T06:37:37.7876713Z - cdk.out/blah-acl-dev.assets.json
2023-09-19T06:37:37.7876958Z - cdk.out/blah-acl-dev.template.json
2023-09-19T06:37:37.7877176Z - jest.config.js
2023-09-19T06:37:37.7877397Z - lib/blah-cdk-stack-stack.d.ts
2023-09-19T06:37:37.7882210Z - lib/blah-cdk-stack-stack.js
2023-09-19T06:37:37.7882486Z - lib/blah-cdk-stack-stack.ts
2023-09-19T06:37:37.7882690Z - package-lock.json
2023-09-19T06:37:37.7882996Z - package.json
2023-09-19T06:37:37.7883197Z - test/blah-cdk-stack.test.d.ts
2023-09-19T06:37:37.7883421Z - test/blah-cdk-stack.test.js
2023-09-19T06:37:37.7883642Z - test/blah-cdk-stack.test.ts
2023-09-19T06:37:37.7883843Z - tsconfig.json
2023-09-19T06:37:37.7884053Z - File extensions: .json, .yaml, .yml
2023-09-19T06:37:37.7884439Z Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')
2023-09-19T06:37:37.7898372Z Popen(['git', 'ls-files', '--exclude-standard', '--ignored', '--others', '--cached'], cwd=/tmp/lint, universal_newlines=False, shell=None, istream=None)
2023-09-19T06:37:38.0633638Z - Excluding .gitignored files [10772]: /tmp/lint/bin/blah-cdk-stack.d.ts, /tmp/lint/bin/blah-cdk-stack.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v2-handler.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v3-handler.bundled.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/index.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/shared.js, /tmp/lint/cdk.out/asset.b992a5b1efd6715c28d74eda79615c6b4c3cf54e6364f2b2166dacf0f6c68602/update-blah-ip-set.py, /tmp/lint/cdk.out/cdk.out, /tmp/lint/cdk.out/manifest.json, /tmp/lint/cdk.out/tree.json, /tmp/lint/cdk.out/blah-acl-dev.assets.json, /tmp/lint/cdk.out/blah-acl-dev.template.json, /tmp/lint/lib/blah-cdk-stack-stack.d.ts, /tmp/lint/lib/blah-cdk-stack-stack.js, /tmp/lint/node_modules/.bin/acorn, /tmp/lint/node_modules/.bin/browserslist, /tmp/lint/node_modules/.bin/cdk, /tmp/lint/node_modules/.bin/esparse, /tmp/lint/node_modules/.bin/esvalidate, /tmp/lint/node_modules/.bin/import-local-fixture, /tmp/lint/node_modules/.bin/jest, /tmp/lint/node_modules/.bin/js-yaml,
...
...
 /tmp/lint/node_modules/yn/readme.md, /tmp/lint/node_modules/yocto-queue/index.d.ts, /tmp/lint/node_modules/yocto-queue/index.js, /tmp/lint/node_modules/yocto-queue/license, /tmp/lint/node_modules/yocto-queue/package.json, /tmp/lint/node_modules/yocto-queue/readme.md, /tmp/lint/test/blah-cdk-stack.test.d.ts, /tmp/lint/test/blah-cdk-stack.test.js
2023-09-19T06:37:38.1360909Z Kept [13] files on [33] found files
2023-09-19T06:37:38.1361164Z Kept files before applying linter filters:
2023-09-19T06:37:38.1361588Z - .checkov.yml
2023-09-19T06:37:38.1361827Z - .devcontainer/devcontainer.json
2023-09-19T06:37:38.1362059Z - .vscode/settings.json
2023-09-19T06:37:38.1362260Z - azure-pipelines.yml
2023-09-19T06:37:38.1362469Z - cdk.context.json
2023-09-19T06:37:38.1362662Z - cdk.json
2023-09-19T06:37:38.1362867Z - cdk.out/manifest.json
2023-09-19T06:37:38.1363184Z - cdk.out/tree.json
2023-09-19T06:37:38.1363410Z - cdk.out/blah-acl-dev.assets.json
2023-09-19T06:37:38.1363771Z - cdk.out/blah-acl-dev.template.json
2023-09-19T06:37:38.1363979Z - package-lock.json
2023-09-19T06:37:38.1364257Z - package.json
2023-09-19T06:37:38.1364430Z - tsconfig.json
2023-09-19T06:37:38.1365290Z [Filters] {'name': 'CLOUDFORMATION_CFN_LINT', 'filter_regex_include': None, 'filter_regex_exclude_descriptor': None, 'filter_regex_exclude_linter': None, 'files_sub_directory': None, 'lint_all_files': False, 'lint_all_other_linters_files': False, 'file_extensions': ['.yml', '.yaml', '.json'], 'file_names_regex': [], 'file_names_not_ends_with': [], 'file_contains_regex': ['AWSTemplateFormatVersion', '(AWS|Alexa|Custom)::'], 
'file_contains_regex_extensions': []}
2023-09-19T06:37:38.1366255Z CLOUDFORMATION_CFN_LINT linter kept 4 files after applying linter filters:
2023-09-19T06:37:38.1366499Z - cdk.out/manifest.json
2023-09-19T06:37:38.1366672Z - cdk.out/tree.json
2023-09-19T06:37:38.1366875Z - cdk.out/blah-acl-dev.assets.json
2023-09-19T06:37:38.1367092Z - cdk.out/blah-acl-dev.template.json
2023-09-19T06:37:38.1367163Z 
2023-09-19T06:37:38.1367427Z +----MATCHING LINTERS-------+------------------+----------------+------------+
2023-09-19T06:37:38.1367644Z | Descriptor     | Linter   | Criteria         | Matching files | Format/Fix |
2023-09-19T06:37:38.1367945Z +----------------+----------+------------------+----------------+------------+
2023-09-19T06:37:38.1368238Z | CLOUDFORMATION | cfn-lint | .yml|.yaml|.json | 4              | no         |
2023-09-19T06:37:38.1368529Z +----------------+----------+------------------+----------------+------------+
2023-09-19T06:37:38.1368711Z ##[endgroup]
2023-09-19T06:37:38.1369160Z [cfn-lint] command: ['cfn-lint', '--config-file', '/action/lib/.automation/.cfnlintrc.yml', 'cdk.out/manifest.json', 'cdk.out/tree.json', 'cdk.out/blah-acl-dev.assets.json', 'cdk.out/blah-acl-dev.template.json']
2023-09-19T06:37:38.1369457Z [cfn-lint] CWD: /tmp/lint
2023-09-19T06:37:44.2570068Z [cfn-lint] result: 14 E1001 Missing top level template section Resources
2023-09-19T06:37:44.2570803Z cdk.out/manifest.json:1:1
2023-09-19T06:37:44.2572702Z 
2023-09-19T06:37:44.2573125Z E1001 Top level template section version is not valid
2023-09-19T06:37:44.2574459Z cdk.out/manifest.json:2:3
2023-09-19T06:37:44.2575075Z 
2023-09-19T06:37:44.2575491Z E1001 Top level template section artifacts is not valid
2023-09-19T06:37:44.2575886Z cdk.out/manifest.json:3:3
...
...

nvuillam · 2023-09-20T09:57:37Z

I think this line helps :)

Excluding .gitignored files [10772]: /tmp/lint/bin/blah-cdk-stack.d.ts, /tmp/lint/bin/blah-cdk-stack.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v2-handler.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/aws-sdk-v3-handler.bundled.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/index.js, /tmp/lint/cdk.out/asset.099a3112386fa620b3069790dcf92c8e64bb5341760a8b0a76bfa843f8ad41df/shared.js, /tmp/lint/cdk.out/asset.b992a5b1efd6715c28d74eda79615c6b4c3cf54e6364f2b2166dacf0f6c68602/update-blah-ip-set.py, /tmp/lint/cdk.out/cdk.out, /tmp/lint/cdk.out/manifest.json, /tmp/lint/cdk.out/tree.json, /tmp/lint/cdk.out/blah-acl-dev.assets.json, /tmp/lint/cdk.out/blah-acl-dev.template.json, /tmp/lint/lib/blah-cdk-stack-stack.d.ts, /tmp/lint/lib/blah-cdk-stack-stack.js, /tmp/lint/node_modules/.bin/acorn, /tmp/lint/node_modules/.bin/browserslist, /tmp/lint/node_modules/.bin/cdk, /tmp/lint/node_modules/.bin/esparse, /tmp/lint/node_modules/.bin/esvalidate, /tmp/lint/node_modules/.bin/import-local-fixture, /tmp/lint/node_modules/.bin/jest, /tmp/lint/node_modules/.bin/js-yaml,
...

I check !

Fixes #2955

nvuillam · 2023-09-20T10:07:02Z

@ashokm I think i got it, PR on the way :)

Fixes #2955

nvuillam · 2023-09-20T21:07:58Z

@ashokm please could confirm it's ok with beta version ?

ashokm · 2023-09-21T06:59:07Z

Hi @nvuillam, I can confirm that the problem no longer exists in the latest beta version and the results are as expected (same as in v6).

Tested beta image:

 - Image Creation Date: 2023-09-20T17:50:24Z
 - Image Revision: 5bcd5083473a8d49bb315fbb45db89c8a6fe7827
 - Image Version: beta

nvuillam · 2023-09-21T08:00:00Z

@ashokm perfect, thanks for the feedback :)
I've been really busy and for the moment I'm the only one handling releases, so there has been none for more than a month, but as I got COVID I have time to manage a new minor release soon, including this fix :)

ashokm · 2023-09-21T08:07:59Z

Hope you have a speedy recovery 🚀

Fixes oxsecurity#2955

burner1024 · 2023-11-19T18:05:03Z

Looks like it's broken again, or was never fixed.

❌ Linted [COPYPASTE] files with [jscpd]: Found 145 error(s) - (62.29s)
- Using [jscpd v3.5.10] https://megalinter.io/7.6.0/descriptors/copypaste_jscpd
- MegaLinter key: [COPYPASTE_JSCPD]
- Rules config: [.jscpd.json]
--Error detail:
Clone found (javascript):
 - .vscode-test/vscode-1.43.2/VSCode-linux-x64/resources/app/out/vs/workbench/contrib/webview/electron-browser/pre/electron-index.js [1:1 - 6:187] (5 lines, 327 tokens)
   .vscode-test/vscode-1.44.2/VSCode-linux-x64/resources/app/out/vs/workbench/contrib/webview/electron-browser/pre/electron-index.js [1:1 - 6:187]

$ grep .vscode .gitignore 
.vscode-test/

nvuillam · 2023-11-19T20:55:07Z

@burner1024 if you want to ignore files with jscpd, you need to update .jscpd.json as this is not a linter that takes a list of files as input :)

https://megalinter.io/beta/configuration/#cli-lint-mode

burner1024 · 2023-11-19T22:16:15Z

I just ran the command and expected IGNORE_GITIGNORED_FILES to be applied to all linters, as the doc suggests.
Is there a list of linters that are ignoring IGNORE_GITIGNORED_FILES setting?

nvuillam · 2023-11-19T22:44:04Z

We could indeed update the documentation to precise that it only applies to linters that works with list of files mode (and not run without list of files on the repository root)

Most of linters handle by themselves to ignore files in .gitignore, but some of them like JSCPD do not, as in the example of our internal jscpd documentation -> https://megalinter.io/latest/descriptors/copypaste_jscpd/

Please feel free to post an issue on jscpd repo to request such enhancement :)

burner1024 · 2023-11-19T22:58:29Z

I see, thank you for explanation.

ashokm added the bug Something isn't working label Sep 14, 2023

nvuillam added a commit that referenced this issue Sep 20, 2023

Fix IGNORE_GITIGNORED_FILES not working anymore

0ad6b10

Fixes #2955

nvuillam mentioned this issue Sep 20, 2023

Fix IGNORE_GITIGNORED_FILES not working anymore #2967

Merged

nvuillam closed this as completed in #2967 Sep 20, 2023

nvuillam added a commit that referenced this issue Sep 20, 2023

Fix IGNORE_GITIGNORED_FILES not working anymore (#2967)

fc6a2e4

Fixes #2955

Kurt-von-Laven mentioned this issue Sep 21, 2023

Not all linters seems to ignore .gitignore files and directories #2846

Closed

sanmai-NL pushed a commit to sanmai-NL/megalinter that referenced this issue Oct 4, 2023

Fix IGNORE_GITIGNORED_FILES not working anymore (oxsecurity#2967)

efe0c48

Fixes oxsecurity#2955

nvuillam mentioned this issue Oct 5, 2023

🫱🏻‍🫲🏽 Merge with MegaLinter ? super-linter/super-linter#4591

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IGNORE_GITIGNORED_FILES not applied for cfn-lint #2955

IGNORE_GITIGNORED_FILES not applied for cfn-lint #2955

ashokm commented Sep 14, 2023 •

edited

nvuillam commented Sep 18, 2023

ashokm commented Sep 19, 2023

nvuillam commented Sep 20, 2023 •

edited

nvuillam commented Sep 20, 2023

nvuillam commented Sep 20, 2023

ashokm commented Sep 21, 2023

nvuillam commented Sep 21, 2023

ashokm commented Sep 21, 2023

burner1024 commented Nov 19, 2023

nvuillam commented Nov 19, 2023

burner1024 commented Nov 19, 2023

nvuillam commented Nov 19, 2023 •

edited

burner1024 commented Nov 19, 2023

IGNORE_GITIGNORED_FILES not applied for cfn-lint #2955

IGNORE_GITIGNORED_FILES not applied for cfn-lint #2955

Comments

ashokm commented Sep 14, 2023 • edited

nvuillam commented Sep 18, 2023

ashokm commented Sep 19, 2023

nvuillam commented Sep 20, 2023 • edited

nvuillam commented Sep 20, 2023

nvuillam commented Sep 20, 2023

ashokm commented Sep 21, 2023

nvuillam commented Sep 21, 2023

ashokm commented Sep 21, 2023

burner1024 commented Nov 19, 2023

nvuillam commented Nov 19, 2023

burner1024 commented Nov 19, 2023

nvuillam commented Nov 19, 2023 • edited

burner1024 commented Nov 19, 2023

ashokm commented Sep 14, 2023 •

edited

nvuillam commented Sep 20, 2023 •

edited

nvuillam commented Nov 19, 2023 •

edited