Proposition of architecture refactoring for Super-Linter V4 (in progress) #791

nvuillam · 2020-10-01T17:44:40Z

Why

After contributing and discussing on this repo issues and PRs, it became clear that it was more and more difficult to maintain and make evolve Super-Linter
I thought that a more modern language would help, and as there is a lot of scripting, Python appeared like a good choice for such requirements, with its simple syntax and heavy community adoption
Here is my proposition of architecture refactoring for Super-Linter V4: a core runtime refactoring, but keeping as much as possible existing elements

Legend:

completed and tested !
🛠️ in progress
to do :)

Targets

ISO capabilities with V3
- Lint files using all V3 variables configuration
- Multi-status: post a sub-status on the PR for each linter
- Write logs in console and log file
- TAP files generation and testing
- Transparent upgrade from v3 to v4 in existing integrations: no need to update configuration
- Build docker image from repo, packaged in Super-Linter GitHub Action
- Validate all code base, or only files diff between current commit and default branch (default: master)
- The possum ;)
Simplify
- Refactoring runtime in Python, for easier handling than bash thanks to classes and python modules
- Everything related to each linter in a single descriptor YML file :
  - easier evolutive maintenance
  - less conflicts to manage between PRs.
  - Few special cases require a python linter class)
- Default behaviours for all linters, with possibility to override part of them for special cases
- Hierarchical architecture: Apply fixes and new behaviours to all linters with a single code update
- Improved file listing management: Collect all linters, then collect all files matching extensions associated with linters, then for each linter set the list of files after applying additional filters (include regex, exclude regex, linter custom filters)
- Have a centralized exclude list (node_modules,.rbenv, etc...)
Improve robustness & stability
- Test classes for each capability
- Test classes for each linter: Automatic generation of test classes using .automation/build.py
- Setup code coverage
- CD / CI
  - Validate multi-status on PR inside each PR (posted from step "Run against all code base")
  - Run test classes and code coverage with pytest during validation GitHub Action
  - Validate descriptor YML files with json schema during build
New features & improvements
- Linters described in YML descriptors
- Simplify languages and linters variables
  - ENABLE = list of languages and formats to apply lint on codebase (default: all)
  - ENABLE_LINTERS = list of linters to apply lint on codebase (default: all)
  - DISABLE = list of languages and formats to skip (default: none)
  - DISABLE_LINTERS = list of linters to skip (default: none)
  - Variables VALIDATE_XXX are still taken in account (but should not be used in association with ENABLE and DISABLE variables)
- Apply include and exclude regexes to a single language or linter: ex: JAVASCRIPT_FILTER_REGEX_INCLUDE
- Improved logs using python terminaltables package and showing linter (and linter version) used and applied filters for each linter processed
- Automatic generation of Dockerfile using YML descriptors
  - Dockerfile commands (FROM, ARG, ENV, COPY, RUN )
  - APK packages (linux)
  - NPM packages (node)
  - PIP packages (python)
  - GEM packages (ruby)
  - Phive packages (PHP)
  - Remove last special cases hardcoded in dockerfile
- Documentation as code
  - Generate linters tables (ordered by type: language, format & tooling format) and include it in README. (see result)
  - Generate one markdown file per Linter, containing all configuration variables, infos and examples (See result)
- 🛠️ Update static documentation
- 🛠️ Cleaning of legacy unused files (delete /lib if PR merged, else if hard-fork keep it to check diff)
- New linters
  - C
  - C++
  - Scala
What we could do in future versions from this new architecture
- Improve again Performances
  - Parallel processing of each linter to improve performances (for v4.1 ?)
  - Call a linter on whole codebase (with filters) instead of one call by file, then analyze output logs to extract each linted file status
- Send super-linter status summary in Github PR comments
- Publish GitHub pages from markdown
- Add jsonschema on schemastore.org (for auto-completion in IDEs)
- Allow to define configuration in a .superlinter file at the root of the the folder
Languages / formats and linters managed

Languages

Language / Format	Linter	Configuration key
BASH	bash-exec	BASH_EXEC
	shellcheck	BASH_SHELLCHECK
	shfmt	BASH_SHFMT
C	cpplint	C_CPPLINT
CLOJURE	clj-kondo	CLOJURE
COFFEE	coffeelint	COFFEE
C++ (CPP)	cpplint	CPP_CPPLINT
C# (CSHARP)	dotnet-format	CSHARP
DART	dartanalyzer	DART
GO	golangci-lint	GO
GROOVY	npm-groovy-lint	GROOVY
JAVA	checkstyle	JAVA
JAVASCRIPT	eslint	JAVASCRIPT_ES
	standard	JAVASCRIPT_STANDARD
JSX	eslint	JSX
KOTLIN	ktlint	KOTLIN
LUA	luacheck	LUA
PERL	perlcritic	PERL
PHP	php	PHP_BUILTIN
	phpcs	PHP_PHPCS
	phpstan	PHP_PHPSTAN
	psalm	PHP_PSALM
POWERSHELL	powershell	POWERSHELL
PYTHON	pylint	PYTHON_PYLINT
	black	PYTHON_BLACK
	flake8	PYTHON_FLAKE8
R	lintr	R
RAKU	raku	RAKU
RUBY	rubocop	RUBY
SCALA	scalafix	SCALA
SQL	sql-lint	SQL
TSX	eslint	TSX
TYPESCRIPT	eslint	TYPESCRIPT_ES
	standard	TYPESCRIPT_STANDARD

Formats

Language / Format	Linter	Configuration key
CSS	stylelint	CSS
ENV	dotenv-linter	ENV
HTML	htmlhint	HTML
JSON	jsonlint	JSON
LATEX	chktex	LATEX
MARKDOWN	markdownlint	MARKDOWN
PROTOBUF	protolint	PROTOBUF
XML	xmllint	XML
YAML	yamllint	YAML

Tooling formats

Language / Format	Linter	Configuration key
ANSIBLE	ansible-lint	ANSIBLE
ARM	arm-ttk	ARM
CLOUDFORMATION	cfn-lint	CLOUDFORMATION
DOCKERFILE	dockerfilelint	DOCKERFILE_DOCKERFILELINT
	hadolint	DOCKERFILE_HADOLINT
EDITORCONFIG	editorconfig-checker	EDITORCONFIG
KUBERNETES	kubeval	KUBERNETES_KUBEVAL
OPENAPI	spectral	OPENAPI
SNAKEMAKE	snakemake	SNAKEMAKE_LINT
	snakefmt	SNAKEMAKE_SNAKEFMT
TERRAFORM	tflint	TERRAFORM_TFLINT
	terrascan	TERRAFORM_TERRASCAN
	terragrunt	TERRAFORM

Questions

Dears Super-Linter maintainers and contributors: (@admiralAwkbar , @GaboFDC , @ferrarimarco ... )
- are you interested in such implementation for V4 ?
- Do you have remarks / questions / propositions to enhance it ?
Dear contributors:
- There are still empty ticks, do not hesitate to post PRs on my fork in branch POC_RefactorInPython, it would confirm that it's easier to add a new linter with this implementation

* Display tests durations * Code coverage increase * + code coverage * Accelerate test cases + raku codecov * Docker build time log

* Display tests durations * Code coverage increase * + code coverage * Accelerate test cases + raku codecov * Docker build time log * Use setup.py for dependencies * Update Dockerfile * ARM + C# + install at descriptor level * Display date more nicely * Dotnet fix * Dotnet fix ";" * try fix arm & dotnet * fixes again * againnnnn * allezzzz * Back * Fix arm * Remove CSharp from this PR * format

* C and C++ Linters * Fix c / c++ tests

* fix version * Update Pipfile * adding it * didnt mean to degrade this * Bump @stoplight/spectral from 5.5.0 to 5.6.0 in /dependencies Bumps [@stoplight/spectral](https://github.com/stoplightio/spectral) from 5.5.0 to 5.6.0. - [Release notes](https://github.com/stoplightio/spectral/releases) - [Changelog](https://github.com/stoplightio/spectral/blob/v5.6.0/CHANGELOG.md) - [Commits](stoplightio/spectral@v5.5.0...v5.6.0) Signed-off-by: dependabot[bot] <support@github.com> * Remove unneeded variables and eval the rest * Initialize all log-related ASAP * Bump yamllint from 1.24.2 to 1.25.0 in /dependencies Bumps [yamllint](https://github.com/adrienverge/yamllint) from 1.24.2 to 1.25.0. - [Release notes](https://github.com/adrienverge/yamllint/releases) - [Changelog](https://github.com/adrienverge/yamllint/blob/master/CHANGELOG.rst) - [Commits](adrienverge/yamllint@v1.24.2...v1.25.0) Signed-off-by: dependabot[bot] <support@github.com> * Bump cfn-lint from 0.36.1 to 0.37.0 in /dependencies Bumps [cfn-lint](https://github.com/aws-cloudformation/cfn-python-lint) from 0.36.1 to 0.37.0. - [Release notes](https://github.com/aws-cloudformation/cfn-python-lint/releases) - [Changelog](https://github.com/aws-cloudformation/cfn-python-lint/blob/master/CHANGELOG.md) - [Commits](aws-cloudformation/cfn-lint@v0.36.1...v0.37.0) Signed-off-by: dependabot[bot] <support@github.com> * Remove unneeded variables from Dockerfile * Correct path to linters directory * Bump snakefmt from 0.2.0 to 0.2.1 in /dependencies Bumps [snakefmt](https://github.com/snakemake/snakefmt) from 0.2.0 to 0.2.1. - [Release notes](https://github.com/snakemake/snakefmt/releases) - [Changelog](https://github.com/snakemake/snakefmt/blob/master/CHANGELOG.md) - [Commits](snakemake/snakefmt@0.2.0...0.2.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump snakemake from 5.25.0 to 5.26.0 in /dependencies Bumps [snakemake](https://github.com/snakemake/snakemake) from 5.25.0 to 5.26.0. - [Release notes](https://github.com/snakemake/snakemake/releases) - [Commits](snakemake/snakemake@v5.25.0...v5.26.0) Signed-off-by: dependabot[bot] <support@github.com> * Bump markdownlint-cli from 0.23.2 to 0.24.0 in /dependencies Bumps [markdownlint-cli](https://github.com/igorshubovych/markdownlint-cli) from 0.23.2 to 0.24.0. - [Release notes](https://github.com/igorshubovych/markdownlint-cli/releases) - [Commits](igorshubovych/markdownlint-cli@v0.23.2...v0.24.0) Signed-off-by: dependabot[bot] <support@github.com> * Dynamically initialize FILE_ARRAY variables * Set FILE_ARRAY_EDITORCONFIG * Populate FILE_ARRAY_DOCKERFILE_HADOLINT * Bump flake8 from 3.8.3 to 3.8.4 in /dependencies Bumps [flake8](https://gitlab.com/pycqa/flake8) from 3.8.3 to 3.8.4. - [Release notes](https://gitlab.com/pycqa/flake8/tags) - [Commits](https://gitlab.com/pycqa/flake8/compare/3.8.3...3.8.4) Signed-off-by: dependabot[bot] <support@github.com> * Bump eslint-plugin-jest from 24.0.2 to 24.1.0 in /dependencies Bumps [eslint-plugin-jest](https://github.com/jest-community/eslint-plugin-jest) from 24.0.2 to 24.1.0. - [Release notes](https://github.com/jest-community/eslint-plugin-jest/releases) - [Changelog](https://github.com/jest-community/eslint-plugin-jest/blob/master/CHANGELOG.md) - [Commits](jest-community/eslint-plugin-jest@v24.0.2...v24.1.0) Signed-off-by: dependabot[bot] <support@github.com> * Remove unnecessary --update for apk in Dockerfile There are two `apk add` use both `--update` and `--no-cache` parameter, which is unnecessary, usually should use only `--no-cache` to prevent leaving cache files, just like the other `apk add` in line 249 * Simplify shfmt install by using prebuilt binary * Bump snakemake from 5.26.0 to 5.26.1 in /dependencies Bumps [snakemake](https://github.com/snakemake/snakemake) from 5.26.0 to 5.26.1. - [Release notes](https://github.com/snakemake/snakemake/releases) - [Commits](snakemake/snakemake@v5.26.0...v5.26.1) Signed-off-by: dependabot[bot] <support@github.com> * Fix documentation error with VALIDATE_MD being renamed to VALIDATE_MARKDOWN * Bump cfn-lint from 0.37.0 to 0.37.1 in /dependencies Bumps [cfn-lint](https://github.com/aws-cloudformation/cfn-python-lint) from 0.37.0 to 0.37.1. - [Release notes](https://github.com/aws-cloudformation/cfn-python-lint/releases) - [Changelog](https://github.com/aws-cloudformation/cfn-python-lint/blob/master/CHANGELOG.md) - [Commits](aws-cloudformation/cfn-lint@v0.37.0...v0.37.1) Signed-off-by: dependabot[bot] <support@github.com> * Bump snakefmt from 0.2.1 to 0.2.2 in /dependencies Bumps [snakefmt](https://github.com/snakemake/snakefmt) from 0.2.1 to 0.2.2. - [Release notes](https://github.com/snakemake/snakefmt/releases) - [Changelog](https://github.com/snakemake/snakefmt/blob/master/CHANGELOG.md) - [Commits](snakemake/snakefmt@0.2.1...0.2.2) Signed-off-by: dependabot[bot] <support@github.com> * Centralize file list building logic in buildFileList.sh * Dynamically load linter rules * adding rules * fix it * fix pip * adding rules * Add support for Terragrunt files * Add Terragrunt Co-authored-by: Lucas Gravley <29484535+admiralAwkbar@users.noreply.github.com> Co-authored-by: Lukas Gravley <admiralawkbar@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Marco Ferrari <ferrari.marco@gmail.com> Co-authored-by: Steven Hunt <st3v3nhunt@users.noreply.github.com> Co-authored-by: Peter Dave Hello <hsu@peterdavehello.org> Co-authored-by: Omeed Musavi <omusavi@microsoft.com> Co-authored-by: Tim Jones <timniverse@gmail.com>

…er MD doc (#46) * Installation info in linters doc * Add descriptor label * :) * Add parent descriptors install in linter doc * First icons :) * Add descriptor icons * Display icons in html to keep max size * Nicer table * Default icon * ready !

* Display tests durations * Code coverage increase * + code coverage * Accelerate test cases + raku codecov * Docker build time log * Use setup.py for dependencies * Update Dockerfile * ARM + C# + install at descriptor level * Display date more nicely * Dotnet fix * Dotnet fix ";" * try fix arm & dotnet * fixes again * againnnnn * allezzzz * Back * Fix arm * Fix c#

…efactorInPython

nvuillam · 2020-10-09T18:43:32Z

Now got a nice doc... with one page per linter, containing links to the linter web site configuration pages...
I just have to fix some stuff with TAP output, and I think my dev can be considered as completed... (up to date list of refactored and new features in the first post ! )

Undepending if this PR will be accepted or if it will end as a hard-fork, I'd love to have some feedback :)

Languages

Language / Format	Linter	Configuration key
BASH	bash-exec	BASH_EXEC
	shellcheck	BASH_SHELLCHECK
	shfmt	BASH_SHFMT
C	cpplint	C_CPPLINT
CLOJURE	clj-kondo	CLOJURE
COFFEE	coffeelint	COFFEE
C++ (CPP)	cpplint	CPP_CPPLINT
C# (CSHARP)	dotnet-format	CSHARP
DART	dartanalyzer	DART
GO	golangci-lint	GO
GROOVY	npm-groovy-lint	GROOVY
JAVA	checkstyle	JAVA
JAVASCRIPT	eslint	JAVASCRIPT_ES
	standard	JAVASCRIPT_STANDARD
JSX	eslint	JSX
KOTLIN	ktlint	KOTLIN
LUA	luacheck	LUA
PERL	perlcritic	PERL
PHP	php	PHP_BUILTIN
	phpcs	PHP_PHPCS
	phpstan	PHP_PHPSTAN
	psalm	PHP_PSALM
POWERSHELL	powershell	POWERSHELL
PYTHON	pylint	PYTHON_PYLINT
	black	PYTHON_BLACK
	flake8	PYTHON_FLAKE8
R	lintr	R
RAKU	raku	RAKU
RUBY	rubocop	RUBY
SCALA	scalafix	SCALA
SQL	sql-lint	SQL
TSX	eslint	TSX
TYPESCRIPT	eslint	TYPESCRIPT_ES
	standard	TYPESCRIPT_STANDARD

Formats

Language / Format	Linter	Configuration key
CSS	stylelint	CSS
ENV	dotenv-linter	ENV
HTML	htmlhint	HTML
JSON	jsonlint	JSON
LATEX	chktex	LATEX
MARKDOWN	markdownlint	MARKDOWN
PROTOBUF	protolint	PROTOBUF
XML	xmllint	XML
YAML	yamllint	YAML

Tooling formats

Language / Format	Linter	Configuration key
ANSIBLE	ansible-lint	ANSIBLE
ARM	arm-ttk	ARM
CLOUDFORMATION	cfn-lint	CLOUDFORMATION
DOCKERFILE	dockerfilelint	DOCKERFILE_DOCKERFILELINT
	hadolint	DOCKERFILE_HADOLINT
EDITORCONFIG	editorconfig-checker	EDITORCONFIG
KUBERNETES	kubeval	KUBERNETES_KUBEVAL
OPENAPI	spectral	OPENAPI
SNAKEMAKE	snakemake	SNAKEMAKE_LINT
	snakefmt	SNAKEMAKE_SNAKEFMT
TERRAFORM	tflint	TERRAFORM_TFLINT
	terrascan	TERRAFORM_TERRASCAN
	terragrunt	TERRAFORM

…ry (#48) * TAP V0 * TAP test cases * Fix tap tests * PR message * Fix post comments * et tu tap tap tap * try several report tap expect file names * Skipped test cases on root avoided * Fix tests reports * pytablewriter in setup.py * ignore ponctuation in tap comparison * tap z y tout * no detail when error ok * Only warning if TAP report messages are different * Fix 'fail_contains' filtering with regexes * Rename openapi test files * Disable terraform TAP report test: error just if file order * Rename action * Fix PR message generation * Update SuperLinter.py * Fix post PR comment * Fix PAT + install comments * quickfix github exception

…le + v3 legacy variables (#50) * v0 * Manage DISABLE_ERRORS * Disable errors on linters + doc * legacy variable ERROR_ON_MISSING_EXEC_BIT * contributing * Fix editorconfig * Use latest packages * Bash exec + build * clojure latest version from bash , not docker image * + libstdc++6 * build * clojure * cloj * clojure fix

zkoppert · 2020-10-12T14:50:22Z

Hi @nvuillam! Great work and especially in such a short amount of time!

The super-linter project is not going to switch over to python as the maintainers are dedicated to continuing it in BASH. I would encourage you to fork this into its own project so the work is not lost. Thanks for valuing the project project enough to put in all this effort!

nvuillam · 2020-10-12T18:25:38Z

Ok, so hard-fork it is, but that's a little bite sad: on one side there will be more users, and on the other side more capabilities...

nvuillam added 30 commits September 19, 2020 14:22

v0

98d75b8

Apply include & exclude filters

ef1df7e

Apply python naming conventions on SuperLinter.py

54e0588

Apply python naming conventions of all py files

cf0c6c8

Base test class

6eab3cd

First almost working test :)

d9e51d2

Working python test.py :)

6660bac

chmod entrypoint

7fbfedc

Dockerfile

2a31ee5

Collect results

3dcc111

Display summary in table

2ae2a1a

Fix Dockerfile

b7f7601

Test cases

d7fc538

Reorganize folder (still issues with imports when from entrypoint)

db996cb

Reorganize folders + Language linter roots + tests + Dockerfile

ca15105

Put entrypoint at root level + test logs

6d62f4b

Pipfile dependencies

5476252

Fix default logging method

56d7ec0

Dockerfile + tests status

3e66f91

Update doc

fd4225d

GITHUB_WORKSPACE variable + tests

45a3bfb

Manage GITHUB_WORKSPACE for CI

8646a87

Test helper

4ea40b4

Python linters

19a81cb

entrypoint.sh

b83716f

gitignore

44684c0

Do not run command when not found

e2ab987

Fix python after linting

5be9657

Entrypoint

8a784a8

Format hyphens + unit tests for all linters

d13d673

nvuillam and others added 11 commits October 8, 2020 12:20

Improve codecov + tests perfs (#42)

b7096b4

* Display tests durations * Code coverage increase * + code coverage * Accelerate test cases + raku codecov * Docker build time log

Add C and C++ linters (#44)

b6377d0

* C and C++ Linters * Fix c / c++ tests

Move Terragrunt in terraform descriptor

c8b3ebb

Descriptor icons

08f70bf

Merge remote-tracking branch 'origin/POC_RefactorInPython' into POC_R…

547065d

…efactorInPython

Regenerate README

6627a2b

Dart icon

e3297a4

nvuillam added 3 commits October 11, 2020 18:44

Rename ENABLE_LANGUAGES into ENABLE (#49)

17017da

nvuillam dismissed a stale review via 714e4e8 October 12, 2020 14:03

super-linter deleted a comment from lorenckrzysztof Oct 12, 2020

super-linter deleted a comment Oct 12, 2020

zkoppert closed this Oct 12, 2020

nvuillam mentioned this pull request Apr 11, 2021

Refactoring superlinter to use a more powerful language #1442

Closed

snyk-bot mentioned this pull request Jul 28, 2021

[Snyk] Upgrade immer from 9.0.3 to 9.0.5 aliscco/super-linter#50

Closed

snyk-bot mentioned this pull request Sep 1, 2021

[Snyk] Security upgrade immer from 9.0.3 to 9.0.6 aliscco/super-linter#107

Closed

This was referenced Nov 1, 2021

Feed jscpd All Files At Once #1619

Closed

Invite To Collaborate With Mega-Linter #2090

Closed

snyk-bot mentioned this pull request Sep 23, 2022

[Snyk] Fix for 10 vulnerabilities aliscco/super-linter#574

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposition of architecture refactoring for Super-Linter V4 (in progress) #791

Proposition of architecture refactoring for Super-Linter V4 (in progress) #791

nvuillam commented Oct 1, 2020 •

edited

Loading

nvuillam commented Oct 9, 2020 •

edited

Loading

zkoppert commented Oct 12, 2020

nvuillam commented Oct 12, 2020

Proposition of architecture refactoring for Super-Linter V4 (in progress) #791

Proposition of architecture refactoring for Super-Linter V4 (in progress) #791

Conversation

nvuillam commented Oct 1, 2020 • edited Loading

Why

Targets

Languages

Formats

Tooling formats

Questions

nvuillam commented Oct 9, 2020 • edited Loading

Languages

Formats

Tooling formats

zkoppert commented Oct 12, 2020

nvuillam commented Oct 12, 2020

nvuillam commented Oct 1, 2020 •

edited

Loading

nvuillam commented Oct 9, 2020 •

edited

Loading