Build Errors

Run automated checks on repositories to improve code quality.

You can install and run it in
Here are instructions on how to fix each error:
Alternatives

Gitlab CI usage

Add this on top of your .gitlab-ci.yml file:

validate:
  image: gramener/builderrors
  script: builderrors

If you used gramex init from gramex before version 1.84, change the following:

Delete .editorconfig, .htmllintrc and .stylelintrc.*
Copy this .eslintrc.yml and run npm install --save-dev eslint eslint-plugin-html eslint-plugin-template
If you have a .flake8 or equivalent, or .bandit file, switch to ruff's pyproject.toml

Github Actions usage

Create a .github/workflows/validate.yml:

name: Run build errors
on: [push, pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: docker://gramener/builderrors

Docker usage

From the folder you want check, run this command on Linux:

docker run --rm -it -v `pwd`:/src gramener/builderrors

On Windows Command Prompt:

docker run --rm -it -v %cd%:/src gramener/builderrors

On Windows PowerShell:

docker run --rm -it -v ${PWD}:/src gramener/builderrors

To log into the container and run commands, use:

docker run --rm -it -v `pwd`:/src gramener/builderrors /bin/sh -l
# Now run `builderrors` or any other command

BitBucket Pipelines usage

To run checks on every push with BitBucket, add this on top of your bitbucket-pipelines.yml:

clone:
  lfs: true

pipelines:
  default:
    - step:
        name: validate
        image: gramener/builderrors
        script:
          - builderrors

Jenkins usage

To run checks on every push with Jenkins pipelines, add this to your Jenkinsfile file:

pipeline {
    stages {
        stage('Build errors') {
            agent {
                docker { image 'gramener/builderrors' }
            }
            steps {
                sh 'builderrors'
            }
        }
    }
}

Local usage

Install Python 3.x
Install Node.js
Install git
Install git-lfs

In bash or Git Bash, from any folder (e.g. C:/projects/) run this:

git clone https://github.com/gramener/builderrors
cd builderrors
bash setup.sh

From the folder you want to test, run this in bash or Git Bash:

bash /wherever-you-installed/builderrors

How to fix install errors:

Cannot uninstall 'PyYAML'. It is a distutils installed project ...: Run pip install --ignore-installed PyYAML first
'bash' is not recognized as an internal or external command: Run in bash or Git bash, not the Command Prompt or PowerShell
flake8: No such file or directory or pyminify: command not found: Ensure you can run python, node and git in the same bash shell, and re-install

It checks only committed files

builderrors only runs on a git repository with committed files.

It WON'T check untracked or .gitignored files.

Options

You can pass options as command-line parameters. For example:

# Report errors only if 100+ lines are duplicated or if Python code lines are over 120 chars
docker run --rm -it -v $(pwd):/src gramener/builderrors \
  builderrors --duplicate-lines=100000 --py-line-length=120

# Same check for local usage
bash /path/to/builderrors --duplicate-lines=100000 --py-line-length=120

Skip checks

To skip specific checks, use --skip=. For example, to skip Flake8 and ESLint, use: --skip=flake8 --skip=eslint.

See List of checks for the full list of checks.

You cannot use --only along with --skip.

Run only specific checks

To run only specific checks, use --only=. For example, to only run Flake8 and ESLint, use: --only=flake8 --only=eslint

See List of checks for the full list of checks.

You cannot use --skip along with --only.

Convert to warnings

To run specific checks as a warning (i.e. report it but don't fail the build), use --warning=. For example, to only warn on Flake8 and ESlint, use --warning=flake8 --warning=eslint

See List of checks for the full list of checks and the default error/warning status.

Convert to errors

To run specific checks as errors (i.e. fail the build if the check fails), use --error=. For example, to raise an error on NPM Audit and PyDoc, use --error=npm-audit --error=pydoc

See List of checks for the full list of checks and the default error/warning status.

Environment variable options

You can also pass options as environment variables. (Command line overrides environment variables.) For example:

# Skip flake8. Report errors only if 100+ lines are duplicated
docker run --rm -it -v $(pwd):/src \
  -e SKIP_FLAKE8=1 -e DUPLICATE_LINES=100 \
  builderrors gramener/builderrors

# Skip Git LFS, eslint and stylelint
SKIP_LFS=1 SKIP_ESLINT=1 SKIP_STYLELINT=1 bash /path/to/builderrors

On Gitlab, set environment variables under Settings > CI / CD > Variables.

List of checks

Here are list of --skip options for checks:

Environment variable	Command line	Type	Meaning
`SKIP_LIB=1`	`--skip=lib`	ERROR	Skip libraries
`SKIP_MINIFIED=1`	`--skip=minified`	ERROR	Skip minified file
`SKIP_LFS=1`	`--skip=lfs`	ERROR	Skip Git LFS
`SKIP_PRETTIER=1`	`--skip=prettier`	ERROR	Skip Prettier
`SKIP_USELESS=1`	`--skip=useless`	ERROR	Skip useless files
`SKIP_DUPLICATE_FILES=1`	`--skip=duplicate-files`	ERROR	Skip duplicate files
`SKIP_DUPLICATE_LINES=1`	`--skip=duplicate-lines`	ERROR	Skip duplicate lines
`SKIP_PY_FILENAMES=1`	`--skip=py-filenames`	ERROR	Skip Python filename
`SKIP_BLACK=1`	`--skip=black`	ERROR	Skip Python Black
`SKIP_FLAKE8=1`	`--skip=flake8`	ERROR	Skip flake8
`SKIP_BANDIT=1`	`--skip=bandit`	ERROR	Skip bandit
`SKIP_ESLINT=1`	`--skip=eslint`	ERROR	Skip eslint
`SKIP_STYLELINT=1`	`--skip=stylelint`	ERROR	Skip stylelint
`SKIP_HTMLHINT=1`	`--skip=htmlhint`	ERROR	Skip htmlhint
`SKIP_NPM_AUDIT=1`	`--skip=npm-audit`	WARNING	Skip npm audit
`SKIP_GITLEAKS=1`	`--skip=gitleaks`	WARNING	Skip gitleaks
`SKIP_FLAKE8_EXTRA=1`	`--skip=flake8-extra`	WARNING	Skip flake8 extra
`SKIP_ESLINT_EXTRA=1`	`--skip=eslint-extra`	WARNING	Skip eslint extra
`SKIP_COMPLEXIY=1`	`--skip=complexity`	WARNING	Skip complexity
`SKIP_DATA_BLOCKS=1`	`--skip=data-blocks`	WARNING	Skip data-blocks
`SKIP_URL_TEMPLATES=1`	`--skip=url-templates`	WARNING	Skip url-templates
`SKIP_PYDOC=1`	`--skip=pydoc`	WARNING	Skip pydoc
`SKIP_ABSOLUTE_URLS=1`	`--skip=absolute-urls`	WARNING	Skip absolute URLs
`SKIP_FOLDERS=1`	`--skip=folders`	INFO	Skip folders
`SKIP_CSS_SIZE=1`	`--skip=css-size`	INFO	Skip CSS size
`SKIP_CODE_SIZE=1`	`--skip=code-size`	INFO	Skip PY/JS size

Replace --skip= or SKIP_ with:

--only= or ONLY_ to run only specific checks
--warning= or WARN_ to convert to warnings
--error= or ERROR_ to convert to errors

Other options include:

Environment variable	Command line	Meaning
`SKIP_ESLINT_DEFAULT=1`	`--skip=eslint-default`	Don't copy `.eslintrc.yml` even if `.eslintrc.*` is missing
`SKIP_STYLELINT_DEFAULT=1`	`--skip=stylelint-default`	Don't copy `.stylelintrc.yml` even if `.stylelintrc.*` is missing
`LFS_SIZE=n`	`--lfs-size=n`	Files over `n` bytes should use Git LFS (default: 1,000,000)
`DUPLICATE_FILESIZE=n`	`--duplicate-filesize=n`	Files over `n` bytes should not be duplicated (default: 100)
`DUPLICATE_LINES=n`	`--duplicate-lines=n`	Duplicate code over `n` lines are not allowed (default: 25)
`PY_LINE_LENGTH=n`	`--py-line-length=n`	Approx line length of Python code used by Black (default: 99)
`MAX_JS_COMPLEXITY=n`	`--max-js-complexity=n`	Report JS functions with `>n` cyclomatic complexity
`MAX_PY_COMPLEXITY=n`	`--max-py-complexity=n`	Report PY functions with `>n` cyclomatic complexity

How to fix errors

builderrors reports these errors:

`lib`

ERROR (lib) don't commit libraries. 15 min

node_modules (or bower_components) should be installed via npm install in each environment

Run git rm -rf node_modules/ bower_components/ to remove the libraries
Add bower_components/ and node_modules/ to your .gitignore
To skip this check, use builderrors --skip=libraries (e.g. to share a git repo for offline installation)

`minified`

ERROR (minified) don't commit minified files. 15 min

Minified files are not source code and shouldn't be version-controlled. They're generated

Run git rm jquery.min.js ...other.min.js to remove the file
Run npm install your-package-name to install the package
Change URLs to point to the package (e.g. node_modules/<lib>/dist/<lib>.min.js)
To skip this check, use builderrors --skip=minified (e.g. if the package is not on npm)

`lfs`

ERROR (lfs) use Git LFS for large files. 15 min

Git stores copies of every version. LFS stores pointers instead

Install Git LFS
Run git lfs install on your repo

For each large file(s), run these commands on bash or Git bash: see help

git rm your-large-file.ext          # Remove and commit
git commit -m"Remove your-large-file.txt"
git lfs track your-large-file.ext   # Use Git LFS for your file(s)
# Copy your-large-file.ext back
git add your-large-file.ext         # Add and commit
git commit -m"Use LFS for your-large-file.txt`

To skip this check, use builderrors --skip=lfs (e.g. if you can't use LFS)

`useless`

ERROR (useless) don't commit useless/generated files. 15 min

Thumbnails (thumbs.db), backups (*~), etc don't need to be committed. Nor logs (*.log)

Run git rm <useless.file> to remove it
Add <useless.file> to your .gitignore
To skip this check, use builderrors --skip=useless (e.g. if you DO need to commit .log files)

`duplicate-files`

ERROR (duplicate-files) delete duplicate files. 10 min/error

You can re-use the same file

Run git rm <duplicate.file> to remove it
Replace <duplicate.file> with the retained file in your code
To allow duplicate files less than 1000 bytes, run builderrors --duplicate-filesize=1000
To skip this check, use builderrors --skip=duplicate-files (e.g. if you need duplicate files for test cases)

`duplicate-lines`

ERROR (duplicate-lines) reduce duplicate lines. 15 min/error

You can re-use the same code

Use loops for code repeated one after another
Use functions for code repeated in different places (either in the same file or different files)
Use function parameters to handle minor variations in the repetition
Use data structures for larger variations. For example, create an array or dictionary that stores all the parameters that vary. Remember: you can use functions as values
Use function generators for extreme variations in code. Write a function to create and return a new function depending on your variation
Refactor the code and test very carefully. (Unit test cases help here)
To ignore duplicates up to 100 lines, run builderrors --duplicate-lines=100
To skip this check, use builderrors --skip=duplicate-lines (e.g. if you need duplicate code for test cases)

`prettier`

ERROR (prettier) auto-format JS/CSS with Prettier. 2 min

It's important to have consistent formatting for readability. We use prettier.

Use the VS Code Prettier - Code Formatter plugin to auto-format your code.

Don't format HTML templates like Tornado / Lodash. Prettier does not support templates

To auto-fix, run npx prettier@3.2 --write "**/*.{js,mjs,ts,jsx,tsx,vue,css,scss,sass,yml,yaml,md}"
To ignore specific files, add a .prettierignore file (e.g. add *.html)
To ignore specific rules, add a .prettierrc file
To skip this check, use builderrors --skip=prettier (e.g. if you temporarily need the build to pass)

`black`

ERROR (black) auto-format Python with Black. 2 min

It's important to have consistent formatting for readability. We use black for Python files.

Use the VS Code Prettier - Black plugin to auto-format your code.

To auto-fix, run:
- pip install black (one-time)
- black . --skip-string-normalization --line-length=99
To ignore specific rules, add a pyproject.toml file
To skip this check, use builderrors --skip=black (e.g. if you temporarily need the build to pass)

Troubleshooting:

black is not recognized as an internal or external command
- FIX: pip install black
pip install black raises a PermissionError
- FIX: pip install --user black instead of pip install black

`py-filenames`

ERROR (py-filenames) use lower_alpha Python paths. 5 min/error

You can't import a Python file unless it has alphanumeric letters. Using lowercase is the convention.

Rename the Python files using lower case alphanumerics and underscore (_)
To skip this check, use builderrors --skip=py-filenames (e.g. if you won't be importing the module)

`flake8`

ERROR (flake8) fix Python errors. 1 min/error

ruff checks for flake8 errors with these rules:

pyflakes
pycodestyle
flake8-2020: catches misuse of sys.version or sys.version_info
flake8-bugbear: catches potential bugs
flake8-comprehensions: simplifies list/dict comprehensions
flake8-debugger: catches pdb, ipdb and breakpoint()
flake8-print: catches print statements (use logging instead)
pep8-naming: standardizes variable names

Notes:

To run locally, use ruff check --select E,W,F,YTT,B,C4,T10,T20,N
To auto-fix, run ruff check --select E,W,F,YTT,B,C4,T10,T20,N --fix
To ignore a specific line, add # noqa: <error-number> at the end, e.g. print("\n") # noqa: T201
To ignore specific rules, add to ruff's pyproject.toml file
To skip this check, use builderrors --skip=flake8 (e.g. if you temporarily need the build to pass)

`bandit`

ERROR (bandit) fix Python security errors. 30 min/error

ruff checks for Bandit security errors in Python.

Re-write the code based on advice from bandit
To ignore a specific line, add a # noqa: <error-number> at the end
- To retain # nosec for bandit, use # nosec # noqa: <error-number>
To skip this check, use builderrors --skip=bandit (e.g. if there are too many false-positives)

`eslint`

ERROR (eslint) fix JavaScript errors. 10 sec/error

ESLint reports JavaScript errors in JS and HTML files -- including HTML templates.

To auto-fix, run npx eslint --fix
To ignore a specific line, add a // eslint-disable-line at the end
To ignore specific rules, add a .eslintrc.yml based on the default
To skip this check, use builderrors --skip=eslint (e.g. if you temporarily need the build to pass)

Common errors:

'x' is not defined. [Error/no-undef]
- Add in your .js: /* globals x, ... */.
- Add in .eslintrc.yml for libraries like d3, $, _, etc.: "globals": {"d3": "readonly", ...}
'x' is assigned a value but never used. [Error/no-unused-vars]
- Add in your .js: /* exported x, ... */ (or don't assign to the variable)

`data-blocks`

ERROR (data-blocks) move large data to JSON. 5 min/error

Avoid large data blocks in your code. Keep code and data separate.

Move data into JSON files (or CSV or any other data file).

Reference: eslint-plugin-no-data-blocks

`stylelint`

ERROR (stylelint) fix CSS errors. 3 min/error

stylelint reports CSS and SASS errors.

Re-write the code based on advice from stylelint
To ignore a specific line, add a /* stylelint-disable-line */ at the end
To ignore specific rules, add a .stylelintrc.yml file based on the default. For example:
- "selector-no-unknown": null allows styling custom web components
To skip this check, use builderrors --skip=stylelint (e.g. if you're using third-party provided CSS)

`htmlhint`

ERROR (htmlhint) fix HTML errors. 5 min/error

htmlhint checks HTML and reports errors.

Re-write the code based on advice from htmlhint
To ignore specific rules, add a .htmlhintrc file based on the default
To skip this check, use builderrors --skip=htmlhint (e.g. if you're building a Lodash template library)

`gitleaks`

ERROR (gitleaks) don't commit secrets. 30 min/error

gitleaks detects hardcoded passwords, API keys, and tokens in git repos.

To fix these, change the password, key or token and delete the key.

.gitleaksignore file ignores fragments. Use with care.

`js-modules`

WARNING (js-modules) use JavaScript modules. 30 min/error

JavaScript modules are the modern way of including JavaScript.

Change <script> to <script type="module">
Consider using lebab or ES5 to ES6
Test your code!

`npm-audit`

WARNING (npm-audit) avoid unsafe npm packages. 15 min/error

npm audit checks for JavaScript package vulnerabilities.

To auto-fix, run npm audit fix
To upgrade all packages to the latest compatible version, run npm upgrade
Change package versions manually and retry

`flake8-extra`

WARNING (flake8-extra) improve Python code. 5 min/error

ruff checks for additional flake8 errors. These are OPTIONAL but GOOD to fix.

flake8-eradicate: reports commented code
flake8-simplify: suggests code simplifications

You can fix these exactly like flake8 errors

`eslint-extra`

WARNING (eslint-extra) improve JavaScript code. 30 sec/error

ESLint reports JavaScript warnings in JS and HTML files. These are OPTIONAL but GOOD to fix.

`complexity`

WARNING (complexity) review complex code. 120 min/error

Break your code into smaller functions. Use clear function names. Re-use as much as possible.

`url-templates`

WARNING (url-templates) move large data to JSON. 5 min/error

Construct URL params with URLSearchParams not template literals. Instead of:

const url = `path?city=${city}&time=${now}`;

... use:

const url = `path?` + new URLSearchParams({ city, time: now }).toString();

This offers:

Automatic Encoding: Special characters are safely encoded.
Readability: The syntax is cleaner and more readable. It abstracts away the manual construction of the query string.
Error Reduction: Reduces the chance of errors like missing an & or incorrectly encoding values.

Reference: eslint-plugin-no-url-params-template

`pydoc`

WARNING (pydoc) document Python code. 15 min/error

Add a docstring to your public Python functions, classes and methods.

We recommend writing docstrings in Markdown (not ReST) in the Google style guide.

`absolute-urls`

WARNING (absolute-urls) avoid absolute URLs. 10 min/error

Avoid URLs that begin with /, e.g. <a href="/login"> or <img src="/assets/icon.png">. If the application is deployed at a different path (e.g. at https://example.org/app/ instead of https://example.org/), these links will break.

Change URLs to relative paths, e.g. <a href="../login"> or <a href="login"> or <img src="../assets/icon.png">.

`folders`

INFO (folders): review folder structure and files

Commit only necessary files
Group files logically into folders

`css-size`

INFO (css-size): review largest CSS code

Shows the number of lines, words and characters in all CSS/SCSS files.

Reduce CSS code using libraries
To skip this check, use builderrors --skip=css-size

`code-size`

INFO (code-size) review largest PY/JS code.

Shows the number of lines, words and characters in all Python / JavaScript files.

Reduce code using configurations, functions, and libraries
To skip this check, use builderrors --skip=code-size

Alternatives

Comprehensive open-source multi-linting tools are available at:

Super-Linter
MegaLinter Runner

Files

README.md

Latest commit

History

README.md

File metadata and controls

Build Errors

Gitlab CI usage

Github Actions usage

Docker usage

BitBucket Pipelines usage

Jenkins usage

Local usage

It checks only committed files

Options

Skip checks

Run only specific checks

Convert to warnings

Convert to errors

Environment variable options

List of checks

How to fix errors

lib

minified

lfs

useless

duplicate-files

duplicate-lines

prettier

black

py-filenames

flake8

bandit

eslint

data-blocks

stylelint

htmlhint

gitleaks

js-modules

npm-audit

flake8-extra

eslint-extra

complexity

url-templates

pydoc

absolute-urls

folders

css-size

code-size