Various issues

[GrabCAD-Gary]

Hi,

I've been looking at implementing Mega-Linter in our workflows and run into several unexpected issues and questions. I'll raise them here and I can make the necessary bug reports if they turn out to be more than just me getting things wrong.

1. I can't find a way to run spelling checkers as I expect to. Enabling CSPELL alone does nothing because it seems as though it's fed whatever files are found by other linters i.e. if I enable YAML then the .yml files are found and only those files are spell-checked. This seems very odd. Wanting to run spelling checks independantly of other checks seems like a blindingly obvious thing. Is there a way to do this?

2. I have been unable to determine whether checking changed files only relies on GIT_DIFF, VALIDATE_ALL_CODEBASE or both. I have found no way to do it locally, although it works on GitHub correctly. No combination of those settings ever causes M-L to check changed files only when run locally.

3. Linting stops with:

  [Text Reporter] Generated TEXT report: /tmp/lint/report/linters_logs/ERROR-YAML_PRETTIER.log
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/megalinter/run.py", line 15, in <module>
    linter.run()
  File "/megalinter/MegaLinter.py", line 167, in run
    reporter.produce_report()
  File "/megalinter/reporters/ConfigReporter.py", line 103, in produce_report
    vscode_extensions_config = json.load(json_file)
  File "/usr/local/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 2 (char 3)

I had this pinned down to cspell until I found that it happens when I enable YAML as well. I checked every JSON file in the repo and none of them have any errors so this seems to be internal. I tried both dotnet and javascript flavours and both have the same issue.

4. Generated files don't pass linting!! This is true of the .mega-linter.yml for the linters and for the GH action and others. That seems pretty poor.

5. Linting is horribly slow for some operations. I can run GIT, YAML and SPELL and have results in about a minute using the javascript and dotnet flavours. If I run POWERSHELL it's about an hour. As I write I have GIT and JSON enabled. M-L found 1467 JSON files to lint, it's been going for over an hour and last reported

✅ Linted [JSON] files with [v8r] successfully - (2845.17s)
[Text Reporter] Generated TEXT report: /tmp/lint/report/linters_logs/SUCCESS-JSON_V8R.log

towards an hour ago. It's not usable at that rate of progress and no linter should take that long. Again, something more seems wrong especially given that they run in parallel. I did note that Docker prompted that I had files shared with WSL, presumably because I used the WSL2 back-end. I've changed that back to default which stopped Docker prompting but hasn't helped.

6. I'm using VSCode. The name completion mentioned on the configuration page doesn't work for ENABLE_LINTERS and DISABLE_LINTERS. That's because they're not in the schema, which seems like an oversight.

Any guidance you can offer is appreciated. I may be doing things wrong somewhere. I have some settings in my .mega-linter.yml and I have tried juggling them to see if anything changes but haven't detected that it does.

Many thanks.

[GrabCAD-Gary]

UPDATE:
The run mentioned above finshed after somewhere around two hours. The output was then:

✅ Linted [JSON] files with [v8r] successfully - (2845.17s)
[Text Reporter] Generated TEXT report: /tmp/lint/report/linters_logs/SUCCESS-JSON_V8R.log
Unable to get number of errors with regex_sum and ✖ ([0-9]+) problem                 # <===================

### Processed [JSON] files
- Using [eslint-plugin-jsonc v1.6.0] https://ota-meshi.github.io/eslint-plugin-jsonc/
- Mega-Linter key: [JSON_ESLINT_PLUGIN_JSONC]
- Rules config: [.eslintrc-json.json]

It then continued with eslint-plugin-jsonc which finished with

--Error detail:

<--- Last few GCs --->
) [25:0x7fd23fcc3030]  5174193 ms: Mark-sweep (reduce) 992.5 (1010.8) -> 992.5 (1011.0) MB, 1392.6 / 0.0 ms  (+ 4.5 ms in 1 steps since start of marking, biggest step 4.5 ms, walltime since start of marking 1397 ms) (average mu = 0.667, current mu = 0.000) [25:0x7fd23fcc3030]  5175927 ms: Mark-sweep (reduce) 992.8 (1011.0) -> 992.7 (1011.3) MB, 1729.4 / 0.0 ms  (+ 4.1 ms in 1 steps since start of marking, biggest step 4.1 ms, walltime since start of marking 1734 ms) (average mu = 0.476, current mu = 0.000)

<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory

and then the same

❌ Linted [JSON] files with [eslint-plugin-jsonc]: Found 1 error(s) - (5222.07s)
[Text Reporter] Generated TEXT report: /tmp/lint/report/linters_logs/ERROR-JSON_ESLINT_PLUGIN_JSONC.log
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/megalinter/run.py", line 15, in <module>
    linter.run()
  File "/megalinter/MegaLinter.py", line 167, in run
    reporter.produce_report()
  File "/megalinter/reporters/ConfigReporter.py", line 103, in produce_report
    vscode_extensions_config = json.load(json_file)
  File "/usr/local/lib/python3.9/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.9/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 2 (char 3)

😢
Note that that last error happened on GH with SPELL enabled, as well.
...and stopped after removing SPELL.

[nvuillam]

Seems lots of issues... I'm sure we can find solutions for at least most of them :)

What version of mega linter are you using, and how do you run it ? ( Ci ? mega-linter-runner ? )

[GrabCAD-Gary]

npx and V4. GH action. Thanks.

[GrabCAD-Gary]

The latest run also threw errors out of markdown lint:

✅ Linted [MARKDOWN] files with [markdownlint]: Found 812 non blocking error(s) - (13.72s)
[Text Reporter] Generated TEXT report: /tmp/lint/report/linters_logs/ERROR-MARKDOWN_MARKDOWNLINT.log
Unable to get number of errors with regex_sum and ERROR: ([0-9]+) dead links found
Unable to get number of errors with regex_sum and ERROR: ([0-9]+) dead links found
...

It looks like one error line per result, possibly. Around 100 of them. Linting continued successfully from there.

[nvuillam]

I can't find a way to run spelling checkers as I expect to. Enabling CSPELL alone does nothing because it seems as though it's fed whatever files are found by other linters i.e. if I enable YAML then the .yml files are found and only those files are spell-checked. This seems very odd. Wanting to run spelling checks independantly of other checks seems like a blindingly obvious thing. Is there a way to do this?

Usually people want to check the spelling of the files they lint
But if you want to check everything, you may try to add SPELL_CSPELL_CLI_LINT_MODE: project in your .mega-linter.yml file

I have been unable to determine whether checking changed files only relies on GIT_DIFF, VALIDATE_ALL_CODEBASE or both. I have found no way to do it locally, although it works on GitHub correctly. No combination of those settings ever causes M-L to check changed files only when run locally.

As described in documentation, GIT_DIFF is the name of a "linter" that checks if the sources contain unsolved git conflicts markers.
VALIDATE_ALL_CODEBASE : false will make that for most of linters (not the ones with cli_lint_mode == project) will analyze only the files that changed between the current branch and the pull request target branch.

Linting stops with: ...

You probably have a .vscode/extensions.json that contains comments. python default json parser does not like that ... but I think I remember that vscode creates json with comments by default, so I'll find another parser that accepts JSONC. Meanwhile, you can just remove the comments of the workaround :)

Generated files don't pass linting!! This is true of the .mega-linter.yml for the linters and for the GH action and others. That seems pretty poor.

Maybe, that's something to add in the to-do list ^^ But in the reports folder, you can find updated_files that contains the auto-fixes.
If you configure auto-fix, a new commit will automatically be pushed.

Linting is horribly slow for some operations. I can run GIT, YAML and SPELL and have results in about a minute using the javascript and dotnet flavours. If I run POWERSHELL it's about an hour. As I write I have GIT and JSON enabled. M-L found 1467 JSON files to lint, it's been going for over an hour and last reported

v8r checks for each JSON and YAML

• if there is a schema on schemastore.com
• if found, validate the file according to the JSON schema

To enhance performances, you can:

• disable linters or languages
• add linter_name _FILTER_REGEX_EXCLUDE for some linters or languages
• add FILTER_REGEX_EXCLUDE for all linters (maybe on the folder containing all your json files ?)

I'm using VSCode. The name completion mentioned on the configuration page doesn't work for ENABLE_LINTERS and DISABLE_LINTERS. That's because they're not in the schema, which seems like an oversight.

Do you have Red Hat YAML extension ? Defaut VsCode does not manage schemastore.com-based auto-completion

[nvuillam]

About the other issues... it seems that the linter, or the CI machines require too many system resources... you project must be pretty big

What so many json files ? Maybe they are generated ? In that case there is no reason to lint them, you should do something like FILTER_REGEX_EXCLUDE: (path/to/json/files/.*\.json)

Some linters like JSCPD do not care about FILTER_REGEX_EXCLUDE, you must ignore files using their own config file, like explained in jscpd documentation

[GrabCAD-Gary]

Thanks for taking the time to reply. This shouldn't be closed, we're a long way from working yet. I can't re-open it. I presume you have disabled that option. Please re-open it.

Usually people want to check the spelling of the files they lint But if you want to check everything, you may try to add SPELL_CSPELL_CLI_LINT_MODE: project in your .mega-linter.yml file

That's an assumption. I can work with that as a limitation, though. Understanding what it's doing is the first barrier. Thanks.

As described in documentation, GIT_DIFF is the name of a "linter" that checks if the sources contain unsolved git conflicts markers. VALIDATE_ALL_CODEBASE : false will make that for most of linters (not the ones with cli_lint_mode == project) will analyze only the files that changed between the current branch and the pull request target branch.

Yes, I know, but nothing tells me what precisely is being used to determine which files have changed and that is the obvious candidate. VALIDATE_ALL_CODEBASE : false has no visible effect on any of the linters I've tried at all. They still find exactly the same numbers of files and lint them. The difference should be obvious since no files of those types have changed in most cases.

You probably have a .vscode/extensions.json that contains comments. python default json parser does not like that ... but I think I remember that vscode creates json with comments by default, so I'll find another parser that accepts JSONC. Meanwhile, you can just remove the comments of the workaround :)

You're right. Thanks.

Linting is horribly slow for some operations...

v8r checks for each JSON and YAML

* if there is a schema on schemastore.com

* if found, validate the file according to the JSON schema

To enhance performances, you can:

* disable linters or languages

* add _linter_name_ _FILTER_REGEX_EXCLUDE for some linters or languages

* add [FILTER_REGEX_EXCLUDE](https://nvuillam.github.io/mega-linter/configuration/#filter-linted-files) for all linters (maybe on the folder containing all your json files ?)

I have done some of this. Thanks for the pointer regarding some linters needing specific instructions. That's good to know. I also have global exclusions in place for the obvious things. I also thought of giving Docker more memory. I did some researech. Docker, by default, gets 50% of available mem. or 8GB, whichever is the lesser. Mine gets 8GB. That should be more than plenty. Is there some way to tell M-L to use more? Is it limited in some way?

Do you have Red Hat YAML extension ? Defaut VsCode does not manage schemastore.com-based auto-completion

I do.

It doesn't work here. This specific completion doesn't appear, either. None appear in ENABLE_LINTERS. There's a problem somewhere.

The biggest issue I currently have is outside these glitches, though. See here (#810), please.
Update: I may have found the cause of #810. See the bug report.

[GrabCAD-Gary]

Some additional points regarding spelling checks. Spelling checking refuses to run at all unless I enable a language linter to find the files. When a language linter is specified

SPELL_MISSPELL_CLI_LINT_MODE: project
SPELL_CSPELL_CLI_LINT_MODE: project

don't do anything when

ENABLE:
    - SPELL

is set i.e. M-L still lists them as being set to file. They do work when

ENABLE_LINTERS:
    - SPELL_CSPELL
    - SPELL_MISSPELL

are set but, even then, the checkers only appear to check the files found by the language linter. I can work with them together with the language linters. This is just for info.

[nvuillam]

I'll try to have a lookat all that, but please csn you open one issue by problem ?
That way, they can be read, fixed and closed one by one

[GrabCAD-Gary]

👍