Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bandit to CI for security linting #1775

Merged
merged 11 commits into from
Sep 1, 2023

Conversation

strickvl
Copy link
Contributor

Added bandit to the CI to detect medium and high severity warnings about code we write.

(Made some fixes to handle the errors it found -- some were ignored (like hashing algorithms where security was not the intended purpose) and other small fixes were just made.)

For the timeout changes, I read that it's best to choose something that is 1 larger than a multiple of 3, and then beyond that I just chose values that I thought made sense for that particular piece of code.

Pre-requisites

Please ensure you have done the following:

  • I have read the CONTRIBUTING.md document.
  • If my change requires a change to docs, I have updated the documentation accordingly.
  • If I have added an integration, I have updated the integrations table and the corresponding website section.
  • I have added tests to cover my changes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Other (add details above)

@strickvl strickvl added enhancement New feature or request internal To filter out internal PRs and issues dependencies Pull requests that update a dependency file tests Test suite additions or updates labels Aug 31, 2023
@strickvl strickvl requested review from stefannica, schustmi, bcdurak and avishniakov and removed request for stefannica and bcdurak August 31, 2023 18:05
Copy link
Contributor

@avishniakov avishniakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, looks good, one question here: you made all those exclusions here and there to suppress known security vulnerabilities? Why I'm asking: some of them are not hard to fix, like switching from md5 to else and so on.

@strickvl
Copy link
Contributor Author

strickvl commented Sep 1, 2023

@avishniakov good question. The problem with changing the hashing algorithm relates to backwards compatibility. i.e. anyone upgrading to a newer version would have the hashes stored in the database using the old algo but this update would mean we essentially wipe / invalidate their cache (in a way that perhaps does unexpected things). To do it in a graceful way would be fairly involved, I felt, where we'd have to maybe store the hashing algorithm in the database along with the hash itself.

Since none of the places where we're currently using old hashing algorithms involve security, I felt like it was an ok tradeoff. It will prevent us from adding any new places where we're using broken hashing algorithms and we can potentially consider switching out the places where it's ignored going forward as part of a bigger ticket.

@strickvl
Copy link
Contributor Author

strickvl commented Sep 1, 2023

You're right though that the fix is basically replacing md5 with sha256, but that would break things in unpredictable ways, I thought.

Copy link
Contributor

@schustmi schustmi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was the reason for the various different timeout adjustments (5/7/31/#nosec) for the different requests?

@strickvl
Copy link
Contributor Author

strickvl commented Sep 1, 2023

What was the reason for the various different timeout adjustments (5/7/31/#nosec) for the different requests?

@schustmi see the PR description. I didn't want to make it generically long, but that said I can also see that it might be confusing. Not sure what the right way to handle this is. Raw request.get() without a timeout passed in was flagged as a medium risk.

@strickvl strickvl merged commit 3259af9 into develop Sep 1, 2023
67 checks passed
@strickvl strickvl deleted the feature/OSS-2335-bandit-safety-security-checks branch September 1, 2023 08:59
avishniakov added a commit that referenced this pull request Sep 4, 2023
* Add `bandit` to CI for security linting (#1775)

* ignore bandit false positives

* add timeout for requests call

* ignore bandit false positive

* ignore bandit false positives

* ignore bandit false positives

* update as per bandit suggestions

* add bandit as a dependency

* add security checks to lint-unit-test workflow

* add safety to dev dependencies + update version

* loosen safety dependency pin

* remove safety dependency

* Add `mlstacks` compatibility check to CI (#1767)

* add compatibility check to CI

* add compatibility check to earlier in CI

* remove dependency on integration tests from CI

* add compatibility check to earlier in CI

* remove compatibility check from integration tests

---------

Co-authored-by: Safoine El Khabich <34200873+safoinme@users.noreply.github.com>

* extend `StepContext` visibility to materializers (#1769)

* extend StepContext visibility to materializers

* skip output processing on failed step

* PR comments

* Revert GH changes to fix colima bug in macos gh (#1779)

* revert the changes done on gh in macos due to failing bug in colima

* revert the changes done on gh in macos due to failing bug in colima

* reinstall qemu

* Reduce CI runner count (#1777)

* combine setup with unit tests

* combine integration with template tests

* further reduce runner count

* remove dynamic uses

---------

Co-authored-by: Safoine El Khabich <34200873+safoinme@users.noreply.github.com>

* Add auto-population of E2E example from template (#1766)

* add checked-out template

* add setup to action

* auto-update e2e

* add OPT_IN_OUT_EMAIL

* pass email

* let e2e auto populate

* debug

* fix mkdir

* debug

* debug

* update commit command

* use ref_name

* update ref

* add ref to checkout

* Auto-update of template

* update template

* Auto-update of template

* restore full CI

* fix branch changes check

* debug

* restore full CI

---------

Co-authored-by: GitHub Actions <actions@github.com>

* fix step name in ci

---------

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>
Co-authored-by: Safoine El Khabich <34200873+safoinme@users.noreply.github.com>
Co-authored-by: GitHub Actions <actions@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file enhancement New feature or request internal To filter out internal PRs and issues tests Test suite additions or updates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants