Lint Readable

[GalRabin]

@anara123
@yaakovi
@yuvalbenshalom
@bakatzir

Status

In Progress

Related Issues

fixes: https://github.com/demisto/etc/issues/21345
fixes: https://github.com/demisto/etc/issues/22148
fixes: https://github.com/demisto/etc/issues/21750 @bakatzir
fixes: https://github.com/demisto/etc/issues/21569
reference: https://github.com/demisto/etc/issues/22759

Description

Lint command readable output.
Until today the lint only printed the raw output to the screen, due to that an overall optimization required in order to parse the data and supply a parsed output.

Changes:

1. Logging option -v ... -vvvvvv (verbosity levels - logging levels verbose-critical- .. - debug)
2. Logging path option - Until debug level.
3. Docker-sdk, git-python usage for better exceptions handling and output,
4. overall optimizations and code readability.
5. JSON output including all collected data for analytics.
6. Adding the following flags:

• test-xml - export test results as xml.
• json-report - export json report on all results.
• log - save the log to file.

7. Git python pack filtering - using set operations (save 19 sec- - each time).
8. Facts gathering optimization - Pipfile collecting, modules from master, additional packages adding to each package 'test-requirements.txt` in the package.
9. Context manager for mandatory test modules (CommonServerPython etc).
10. Code readability.
11. Docker image builds and not commit (Not the recommended way), Now we are using only Single Docker file to the setup test image.
12. Adding many unit-tests in order to be more resilient, exceptions, fixtures etc - https://github.com/demisto/demisto-sdk/tree/lint/demisto_sdk/commands/lint/tests.

CI example

• When running against branch (no errors) - https://circleci.com/gh/demisto/content/44338#queue-placeholder/containers/1
• When running against branch (pytest error trigered) -
  https://circleci.com/gh/demisto/content/44341#queue-placeholder/containers/1
• When running on all packages (pytest error trigered) - https://app.circleci.com/pipelines/github/demisto/content/4617/workflows/5cb54459-d3b2-482f-b1ac-7f345d2d1676/jobs/44355
  ** We need to provide solution for vulture errors.
  Log example https://44355-60525392-gh.circle-artifacts.com/1/artifacts/lint_debug_log.log (If exception occured-search for critical, error - if lint occurred"
  json report example- https://44355-60525392-gh.circle-artifacts.com/1/artifacts/lint_report.json (Sort - by pack name)
  Test failure presentation of CircleCi- https://app.circleci.com/pipelines/github/demisto/content/4617/workflows/5cb54459-d3b2-482f-b1ac-7f345d2d1676/jobs/44355/tests

Change that should be done in config.yml in content repo:

demisto/content#6099

[fvigo]

During the refactoring, please make sure that the error checks are in place. For example, running demisto-sdk lint on a machine that doesn't have docker returns this error, which is not very clear:

Traceback (most recent call last):
  File "/usr/local/bin/demisto-sdk", line 11, in <module>
    load_entry_point('demisto-sdk==0.4.2', 'console_scripts', 'demisto-sdk')()
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/demisto_sdk-0.4.2-py3.7.egg/demisto_sdk/__main__.py", line 321, in lint
    return linter.run_dev_packages()
  File "/usr/local/lib/python3.7/site-packages/demisto_sdk-0.4.2-py3.7.egg/demisto_sdk/commands/lint/lint_manager.py", line 126, in run_dev_packages
    run_status_code = linter.run_dev_packages()
  File "/usr/local/lib/python3.7/site-packages/demisto_sdk-0.4.2-py3.7.egg/demisto_sdk/commands/lint/linter.py", line 134, in run_dev_packages
    py_num = get_python_version(dockers[0], self.log_verbose)
  File "/usr/local/lib/python3.7/site-packages/demisto_sdk-0.4.2-py3.7.egg/demisto_sdk/commands/common/tools.py", line 453, in get_python_version
    universal_newlines=True, stderr=stderr_out).strip()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'run', '--rm', 'demisto/python3:3.7.4.2245', 'python', '-c', "import sys;print('{}.{}'.format(sys.version_info[0], sys.version_info[1]))"]' returned non-zero exit status 125.

[GalRabin]

Updating on changes done until now:

1. Reducing the size of docker images (75% precent on alpine versions due to my implementation mistake)- Reduce the running time to 37:00 min nightly instead of 43:00 min
2. Verbosity level by default on -vv (by @glicht request)
3. Better user experience - tested on daily job of 12 developers, Recieved many feedbacks from the users regarding the output and I changed it to satisfied their needs.
4. Bugs found in -
   • Powershell analyze and test has been fixed.
   • Git filters has been optimized and tested and diffrent enviorments.
   • Yaml filtering error fixed.

[lgtm-com]

This pull request introduces 1 alert when merging 7de1f47 into 43ce5b0 - view on LGTM.com

new alerts:

• 1 for Unused local variable

[GalRabin]

Nightly circle ci tests (The optimal number of threads to be in use is 8, From tests I perfromed, Due to GIL):

• 8 Threads (37:06 min at night) - https://circleci.com/gh/demisto/content/46415#tests/containers/1
• 8 Threads (36:46 min at night) - https://circleci.com/gh/demisto/content/46344#tests/containers/1
• 8 Threads (42:02 min - Done in the middle of the day) - https://circleci.com/gh/demisto/content/46457#queue-placeholder/containers/1
• 8 Threads (38:00 min) - https://app.circleci.com/pipelines/github/demisto/content/6378/workflows/956d82da-5f6e-4fc0-903f-fd0dbf5c72da/jobs/46470/parallel-runs/1?filterBy=ALL

Build with some errors:

• https://app.circleci.com/pipelines/github/demisto/content/6498/workflows/4144392d-144a-4751-88a1-43d9d239e4e0/jobs/46479/parallel-runs/1?filterBy=FAILED
• https://app.circleci.com/pipelines/github/demisto/content/6504/workflows/c920007c-a190-4af4-9071-2f5370aa2cf8/jobs/46486/parallel-runs/1?filterBy=ALL

Next steps before release:

• Changing config.yml
• Changing pre-commit remove -p from command.

[glicht]

Looks good. Happy to see this moving forward.

Please review unit test coverage as this PR decreases coverage.