fix(general): fix multiprocess abilities #5887

AdamDev · 2023-12-20T11:20:12Z

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

There are 2 bugs:

SCA and SAST are not working well in Darwin via Fork mode (security issue of Mac).
Can't override Windows parallel type to None.

Solution:

In the case of Darwin, change the parallel type to Threads, the best practice is 'spawn ' but it starts the child process with a fresh memory, and Checkov is not supporting this case yet, the runners still use the parent's memory.
Support ability for None parallel type in Windows.

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my feature, policy, or fix is effective and works
New and existing tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules

gruebel

feels like cheating by excluding the secrets runner, which failed the last time 😄

checkov/common/parallelizer/parallel_runner.py

checkov/kustomize/runner.py

checkov/common/parallelizer/parallel_runner.py

gruebel

🍰

* fix_multiprocess_ability_in_mac * support freeze mode * update description * lint * flake8 fix * testing * support secret scan * lint * support secret * get parallel type to execute * typo * flake 8 * clean * improve comment * moving back to threads in casd of Darwin * clean * revert some changes * revert some changes * revert changes * clean * clean * clean * add comments * add UT * allow to override via env var * re-run --------- Co-authored-by: adam varsano <avarsano@paloaltonetworks.com>

adam varsano added 2 commits December 19, 2023 17:19

fix_multiprocess_ability_in_mac

7ced7fb

support freeze mode

1ca50d1

AdamDev had a problem deploying to scan-security December 20, 2023 11:20 — with GitHub Actions Failure

update description

9d96d85

AdamDev had a problem deploying to scan-security December 20, 2023 11:21 — with GitHub Actions Failure

lint

88e731d

AdamDev had a problem deploying to scan-security December 20, 2023 11:36 — with GitHub Actions Failure

flake8 fix

432d9b4

AdamDev had a problem deploying to scan-security December 20, 2023 11:44 — with GitHub Actions Failure

testing

847e766

AdamDev had a problem deploying to scan-security December 20, 2023 13:52 — with GitHub Actions Failure

support secret scan

98a9803

AdamDev had a problem deploying to scan-security December 20, 2023 15:08 — with GitHub Actions Failure

lint

a405088

AdamDev had a problem deploying to scan-security December 20, 2023 15:12 — with GitHub Actions Failure

support secret

77a1625

AdamDev had a problem deploying to scan-security December 20, 2023 16:06 — with GitHub Actions Failure

get parallel type to execute

2256b3a

AdamDev had a problem deploying to scan-security December 20, 2023 18:59 — with GitHub Actions Failure

typo

80ff14b

AdamDev had a problem deploying to scan-security December 20, 2023 19:05 — with GitHub Actions Failure

flake 8

315b461

AdamDev had a problem deploying to scan-security December 20, 2023 19:09 — with GitHub Actions Failure

clean

5996bda

AdamDev had a problem deploying to scan-security December 21, 2023 06:03 — with GitHub Actions Failure

improve comment

99190d7

AdamDev had a problem deploying to scan-security December 21, 2023 06:05 — with GitHub Actions Failure

gruebel reviewed Dec 21, 2023

View reviewed changes

checkov/common/parallelizer/parallel_runner.py Outdated Show resolved Hide resolved

checkov/common/parallelizer/parallel_runner.py Outdated Show resolved Hide resolved

checkov/kustomize/runner.py Outdated Show resolved Hide resolved

moving back to threads in casd of Darwin

efb78a8

AdamDev had a problem deploying to scan-security December 24, 2023 06:54 — with GitHub Actions Failure

AdamDev had a problem deploying to scan-security December 24, 2023 07:11 — with GitHub Actions Failure

revert some changes

53bcaba

AdamDev had a problem deploying to scan-security December 24, 2023 07:15 — with GitHub Actions Failure

revert some changes

91eb218

AdamDev had a problem deploying to scan-security December 24, 2023 07:16 — with GitHub Actions Failure

revert changes

1767f63

AdamDev had a problem deploying to scan-security December 24, 2023 07:19 — with GitHub Actions Failure

clean

49d5c6a

AdamDev had a problem deploying to scan-security December 24, 2023 07:20 — with GitHub Actions Failure

clean

ac5177a

AdamDev had a problem deploying to scan-security December 24, 2023 07:20 — with GitHub Actions Failure

adam varsano added 2 commits December 24, 2023 10:21

clean

0092bef

add comments

0ee5a2c

AdamDev had a problem deploying to scan-security December 24, 2023 09:21 — with GitHub Actions Failure

add UT

5df185b

AdamDev temporarily deployed to scan-security December 24, 2023 10:22 — with GitHub Actions Inactive

AdamDev requested a review from gruebel December 24, 2023 10:29

gruebel reviewed Dec 25, 2023

View reviewed changes

checkov/common/parallelizer/parallel_runner.py Show resolved Hide resolved

allow to override via env var

5f271b9

AdamDev had a problem deploying to scan-security December 25, 2023 16:23 — with GitHub Actions Failure

AdamDev requested a review from gruebel December 25, 2023 16:32

gruebel approved these changes Jan 4, 2024

View reviewed changes

SteveVaknin approved these changes Jan 9, 2024

View reviewed changes

achiar99 approved these changes Jan 9, 2024

View reviewed changes

re-run

58a5e1a

AdamDev temporarily deployed to scan-security January 10, 2024 05:27 — with GitHub Actions Inactive

AdamDev merged commit d912b2d into main Jan 10, 2024
37 checks passed

AdamDev deleted the fix_multiprocess_ability_in_mac branch January 10, 2024 05:37

Saarett mentioned this pull request Jan 16, 2024

Exception thrown on MAC when running checkov pre-commit hook 3.1.60 #5945

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(general): fix multiprocess abilities #5887

fix(general): fix multiprocess abilities #5887

AdamDev commented Dec 20, 2023 •

edited

Loading

gruebel left a comment

gruebel left a comment

fix(general): fix multiprocess abilities #5887

fix(general): fix multiprocess abilities #5887

Conversation

AdamDev commented Dec 20, 2023 • edited Loading

Checklist:

gruebel left a comment

Choose a reason for hiding this comment

gruebel left a comment

Choose a reason for hiding this comment

AdamDev commented Dec 20, 2023 •

edited

Loading