Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New output #1320

Merged
merged 67 commits into from
Aug 3, 2023
Merged
Show file tree
Hide file tree
Changes from 66 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
fac2f7b
phase-1
Jul 6, 2023
f54bfab
factory
Jul 10, 2023
6b643f3
wip: feat(cli): add an image scanning command
vladklokun Jul 10, 2023
614c5ea
wip: feat: add image scanning service
vladklokun Jul 10, 2023
5eb9c29
chore: include dependencies
vladklokun Jul 10, 2023
dcfcc1d
wip: adjust image scanning service
vladklokun Jul 10, 2023
4253005
wip: feat: use scanning service in CLI
vladklokun Jul 10, 2023
4ecd366
use iface
Jul 10, 2023
5752216
touches
Jul 11, 2023
46e1cea
Merge branch 'feat-image-scan-svc' into new-output
Jul 11, 2023
ad0103c
continue
Jul 17, 2023
ab634de
add cmd
Jul 17, 2023
e1f7e06
Merge branch 'master' of github.com:armosec/kubescape into scan-workload
amirmalka Jul 20, 2023
557c063
support single workload scan
amirmalka Jul 20, 2023
eab88f0
Merge branch 'scan-workload' of github.com:amirmalka/kubescape into s…
amirmalka Jul 20, 2023
7f812f2
fix conflict
amirmalka Jul 20, 2023
5c6e66d
Merge branch 'scan-workload' of github.com:amirmalka/kubescape into n…
Jul 25, 2023
315fe79
Merge branch 'master' into new-output
Jul 25, 2023
219b32d
identifiers
Jul 25, 2023
03b69bf
go mod
Jul 25, 2023
e67fcae
feat(imagescan): add an image scanning command
vladklokun Jul 10, 2023
755d8c3
Merge branch 'master' of github.com:amirmalka/kubescape into scan-wor…
amirmalka Jul 25, 2023
3177ab4
chore(imagescan): include dependencies
vladklokun Jul 10, 2023
3c32003
chore(imagescan): add dependencies to httphandler
vladklokun Jul 19, 2023
0df62cb
Merge branch 'master' of github.com:amirmalka/kubescape into scan-wor…
amirmalka Jul 25, 2023
aab9cd5
added unit tests
amirmalka Jul 25, 2023
6341947
merge
Jul 25, 2023
811914a
more
Jul 25, 2023
dd3b5bf
Merge branch 'feat-image-scan-svc' of github.com:kubescape/kubescape …
Jul 25, 2023
6679ac5
Merge branch 'feat-image-scan-svc' into new-output
Jul 25, 2023
e57d655
integrate img scan
Jul 26, 2023
facbc47
added unit tests
amirmalka Jul 25, 2023
c98b696
more refactoring
amirmalka Jul 27, 2023
6cefd56
add scanned workload reference to opasessionobj
amirmalka Jul 27, 2023
5b61611
fix GetWorkloadParentKind
amirmalka Jul 27, 2023
27482a9
Merge remote-tracking branch 'other/scan-workload' into new-output
Jul 27, 2023
2121b20
Merge remote-tracking branch 'other/scan-workload' into new-output
Jul 27, 2023
d740ba3
remove namespace argument from pullSingleResource, using field select…
amirmalka Jul 27, 2023
e972df9
removed designators (unused) field from PolicyIdentifier, and designa…
amirmalka Jul 27, 2023
05cb1ab
Merge remote-tracking branch 'other/scan-workload' into new-output
Jul 30, 2023
e794ab8
changes
Jul 30, 2023
dd8f624
changes
Jul 31, 2023
e4bafad
fixes
Jul 31, 2023
e829af7
changes
Jul 31, 2023
c224c91
feat(imagescan): add an image scanning command
vladklokun Jul 10, 2023
1dd98f4
chore(imagescan): include dependencies
vladklokun Jul 10, 2023
18fe0a9
chore(imagescan): add dependencies to httphandler
vladklokun Jul 19, 2023
b3f6648
fixes
Aug 1, 2023
34a52b4
chore(imagescan): create vuln db with dedicated function
vladklokun Aug 1, 2023
fe8eb01
docs(imagescan): provide package-level docs
vladklokun Aug 1, 2023
9743a81
Merge branch 'feat-image-scan-svc' into new-output
Aug 1, 2023
5a22fe6
finish merge
Aug 1, 2023
e73538f
Merge branch 'master' into new-output
Aug 1, 2023
4b6f181
image scan tests
Aug 1, 2023
691e329
Merge branch 'master' into new-output
Aug 1, 2023
3d558dd
continue
Aug 1, 2023
30aa191
fixes
Aug 2, 2023
7e62f8e
refactor
Aug 2, 2023
080e929
Merge branch 'master' into new-output
Aug 2, 2023
754dc2e
rm duplicate
Aug 2, 2023
8f90426
start fixes
Aug 2, 2023
bf34b05
update gh actions
dwertent Aug 2, 2023
7ecfee8
Merge pull request #1324 from dwertent/new-output
dwertent Aug 2, 2023
141030c
pr fixes
Aug 3, 2023
86b1f4e
fix test
Aug 3, 2023
80c94cb
Merge branch 'new-output' of github.com:kubescape/kubescape into new-…
Aug 3, 2023
e02304c
improvements
Aug 3, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
20 changes: 16 additions & 4 deletions .github/workflows/00-pr-scanner.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,9 @@ name: 00-pr_scanner
on:
pull_request:
types: [opened, reopened, synchronize, ready_for_review]
branches:
- 'master'
- 'main'
- 'dev'
paths-ignore:
- '**.yaml'
- '**.yml'
- '**.md'
- '**.sh'
- 'website/*'
Expand All @@ -29,3 +26,18 @@ jobs:
RELEASE: ""
CLIENT: test
secrets: inherit

binary-build:
if: ${{ contains( github.event.pull_request.labels.*.name, 'trigger-integration-test') }} ## run only if labeled as "trigger-integration-test"
uses: ./.github/workflows/b-binary-build-and-e2e-tests.yaml
needs: pr-scanner
with:
COMPONENT_NAME: kubescape
CGO_ENABLED: 1
GO111MODULE: ""
GO_VERSION: "1.20"
RELEASE: ""
CLIENT: test
ARCH_METRIX: ""
OS_METRIX: "ubuntu-20.04"
secrets: inherit
34 changes: 0 additions & 34 deletions .github/workflows/01-pr-merged.yaml

This file was deleted.

12 changes: 0 additions & 12 deletions .github/workflows/a-pr-scanner.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,3 @@ jobs:
- Credentials scan: ${{ steps.credentials-scan.outcome }}
- Vulnerabilities scan: ${{ steps.vulnerabilities-scan.outcome }}
reactions: 'eyes'
basic-tests:
needs: scanners
uses: ./.github/workflows/b-binary-build-and-e2e-tests.yaml
with:
COMPONENT_NAME: kubescape
CGO_ENABLED: 1
GO111MODULE: ""
GO_VERSION: "1.20"
RELEASE: ${{ inputs.RELEASE }}
CLIENT: ${{ inputs.CLIENT }}
CHECKOUT_REPO: ${{ github.repository }}
secrets: inherit
74 changes: 66 additions & 8 deletions .github/workflows/b-binary-build-and-e2e-tests.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,45 @@
name: b-binary-build-and-e2e-tests

on:
workflow_dispatch:
inputs:
COMPONENT_NAME:
required: false
type: string
default: "kubescape"
RELEASE:
required: false
type: string
default: ""
CLIENT:
required: false
type: string
default: "test"
GO_VERSION:
required: false
type: string
default: "1.20"
GO111MODULE:
required: false
type: string
default: ""
CGO_ENABLED:
type: number
default: 1
required: false
OS_METRIX:
type: string
required: false
default: '[ "ubuntu-20.04", "macos-latest", "windows-latest"]'
ARCH_METRIX:
type: string
required: false
default: '[ "", "arm64"]'
BINARY_TESTS:
type: string
required: false
default: '[ "scan_nsa", "scan_mitre", "scan_with_exceptions", "scan_repository", "scan_local_file", "scan_local_glob_files", "scan_local_list_of_files", "scan_nsa_and_submit_to_backend", "scan_mitre_and_submit_to_backend", "scan_local_repository_and_submit_to_backend", "scan_repository_from_url_and_submit_to_backend", "scan_with_exception_to_backend", "scan_with_custom_framework", "scan_customer_configuration", "host_scanner", "scan_compliance_score" ]'

workflow_call:
inputs:
COMPONENT_NAME:
Expand All @@ -23,18 +63,22 @@ on:
BINARY_TESTS:
type: string
default: '[ "scan_nsa", "scan_mitre", "scan_with_exceptions", "scan_repository", "scan_local_file", "scan_local_glob_files", "scan_local_list_of_files", "scan_nsa_and_submit_to_backend", "scan_mitre_and_submit_to_backend", "scan_local_repository_and_submit_to_backend", "scan_repository_from_url_and_submit_to_backend", "scan_with_exception_to_backend", "scan_with_custom_framework", "scan_customer_configuration", "host_scanner", "scan_compliance_score" ]'
CHECKOUT_REPO:
OS_METRIX:
type: string
required: false
default: '[ "ubuntu-20.04", "macos-latest", "windows-latest"]'
ARCH_METRIX:
type: string



required: false
default: '[ "", "arm64"]'
jobs:
wf-preparation:
name: secret-validator
runs-on: ubuntu-latest
outputs:
TEST_NAMES: ${{ steps.export_tests_to_env.outputs.TEST_NAMES }}
OS_METRIX: ${{ steps.export_os_to_env.outputs.OS_METRIX }}
ARCH_METRIX: ${{ steps.export_arch_to_env.outputs.ARCH_METRIX }}
is-secret-set: ${{ steps.check-secret-set.outputs.is-secret-set }}
steps:
- name: check if the necessary secrets are set in github secrets
Expand All @@ -49,32 +93,46 @@ jobs:
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: "echo \"is-secret-set=${{ env.CUSTOMER != '' && \n env.USERNAME != '' &&\n env.PASSWORD != '' &&\n env.CLIENT_ID != '' &&\n env.SECRET_KEY != '' &&\n env.REGISTRY_USERNAME != '' &&\n env.REGISTRY_PASSWORD != ''\n }}\" >> $GITHUB_OUTPUT\n"

- id: export_os_to_env
name: set test name
run: |
echo "OS_METRIX=$input" >> $GITHUB_OUTPUT
env:
input: ${{ inputs.OS_METRIX }}

- id: export_tests_to_env
name: set test name
run: |
echo "TEST_NAMES=$input" >> $GITHUB_OUTPUT
env:
input: ${{ inputs.BINARY_TESTS }}


- id: export_arch_to_env
name: set test name
run: |
echo "ARCH_METRIX=$input" >> $GITHUB_OUTPUT
env:
input: ${{ inputs.ARCH_METRIX }}


binary-build:
name: Create cross-platform build
needs: wf-preparation
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GOARCH: ${{ matrix.arch }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-20.04, macos-latest, windows-latest]
arch: ["", arm64]
os: ${{ fromJson(needs.wf-preparation.outputs.OS_METRIX) }}
arch: ${{ fromJson(needs.wf-preparation.outputs.ARCH_METRIX) }}
exclude:
- os: windows-latest
arch: arm64
steps:

- uses: actions/checkout@ac593985615ec2ede58e132d2e21d2b1cbd6127c # ratchet:actions/checkout@v3
with:
repository: ${{inputs.CHECKOUT_REPO}}
fetch-depth: 0
submodules: recursive

Expand Down
7 changes: 5 additions & 2 deletions cmd/scan/framework.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,14 +78,15 @@ func getFrameworkCmd(ks meta.IKubescape, scanInfo *cautils.ScanInfo) *cobra.Comm

var frameworks []string

if len(args) == 0 { // scan all frameworks
if len(args) == 0 {
Daniel-GrunbergerCA marked this conversation as resolved.
Show resolved Hide resolved
scanInfo.ScanAll = true
} else {
// Read frameworks from input args
frameworks = strings.Split(args[0], ",")
if cautils.StringInSlice(frameworks, "all") != cautils.ValueNotFound {
scanInfo.ScanAll = true
frameworks = getter.NativeFrameworks

}
if len(args) > 1 {
if len(args[1:]) == 0 || args[1] != "-" {
Expand All @@ -105,6 +106,7 @@ func getFrameworkCmd(ks meta.IKubescape, scanInfo *cautils.ScanInfo) *cobra.Comm
}
}
}
scanInfo.SetScanType(cautils.ScanTypeFramework)
scanInfo.FrameworkScan = true

scanInfo.SetPolicyIdentifiers(frameworks, apisv1.KindFramework)
Expand All @@ -118,7 +120,8 @@ func getFrameworkCmd(ks meta.IKubescape, scanInfo *cautils.ScanInfo) *cobra.Comm
if err = results.HandleResults(ctx); err != nil {
logger.L().Fatal(err.Error())
}
if !scanInfo.VerboseMode {

if !scanInfo.VerboseMode && scanInfo.ScanType == cautils.ScanTypeFramework {
logger.L().Info("Run with '--verbose'/'-v' flag for detailed resources view\n")
}
if results.GetRiskScore() > float32(scanInfo.FailThreshold) {
Expand Down
25 changes: 20 additions & 5 deletions cmd/scan/image.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ package scan
import (
"context"
"fmt"
"os"

logger "github.com/kubescape/go-logger"
"github.com/kubescape/kubescape/v2/core/cautils"
"github.com/kubescape/kubescape/v2/core/core"
"github.com/kubescape/kubescape/v2/core/meta"
"github.com/kubescape/kubescape/v2/core/pkg/resultshandling"
"github.com/kubescape/kubescape/v2/pkg/imagescan"

"github.com/anchore/grype/grype/presenter"
"github.com/spf13/cobra"
)

Expand Down Expand Up @@ -58,15 +58,30 @@ func getImageCmd(ks meta.IKubescape, scanInfo *cautils.ScanInfo, imgScanInfo *im
}

userInput := args[0]

logger.L().Info(fmt.Sprintf("Scanning image: %s", userInput))
scanResults, err := svc.Scan(ctx, userInput, creds)
if err != nil {
return err
}
logger.L().Success("Image scan completed successfully")

scanInfo.SetScanType(cautils.ScanTypeImage)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to pre-run


outputPrinters := core.GetOutputPrinters(scanInfo, ctx)

presenterConfig, _ := presenter.ValidatedConfig("table", "", false)
pres := presenter.GetPresenter(presenterConfig, *scanResults)
uiPrinter := core.GetUIPrinter(ctx, scanInfo)

resultsHandler := resultshandling.NewResultsHandler(nil, outputPrinters, uiPrinter)

resultsHandler.ImageScanData = []cautils.ImageScanData{
{
PresenterConfig: scanResults,
Image: userInput,
},
}

pres.Present(os.Stdout)
resultsHandler.HandleResults(ctx)

if imagescan.ExceedsSeverityThreshold(scanResults, failOnSeverity) {
terminateOnExceedingSeverity(scanInfo, logger.L())
Expand Down
41 changes: 40 additions & 1 deletion cmd/scan/scan.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package scan

import (
"context"
"flag"
"fmt"
"strings"
Expand All @@ -9,6 +10,7 @@ import (
"github.com/kubescape/kubescape/v2/core/cautils"
"github.com/kubescape/kubescape/v2/core/cautils/getter"
"github.com/kubescape/kubescape/v2/core/meta"
v1 "github.com/kubescape/opa-utils/httpserver/apis/v1"
"github.com/spf13/cobra"
)

Expand Down Expand Up @@ -41,14 +43,20 @@ func GetScanCommand(ks meta.IKubescape) *cobra.Command {
Long: `The action you want to perform`,
Example: scanCmdExamples,
Args: func(cmd *cobra.Command, args []string) error {
if len(args) > 0 {
// setting input patterns for framework scan is only relevancy for non-security view
if len(args) > 0 && !scanInfo.IsSecurityView {
if args[0] != "framework" && args[0] != "control" {
return getFrameworkCmd(ks, &scanInfo).RunE(cmd, append([]string{strings.Join(getter.NativeFrameworks, ",")}, args...))
}
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
if scanInfo.IsSecurityView {
setSecurityViewScanInfo(args, &scanInfo)

return securityScan(scanInfo, ks)
}

if len(args) == 0 {
return getFrameworkCmd(ks, &scanInfo).RunE(cmd, []string{strings.Join(getter.NativeFrameworks, ",")})
Expand Down Expand Up @@ -90,6 +98,8 @@ func GetScanCommand(ks meta.IKubescape) *cobra.Command {
scanCmd.PersistentFlags().BoolVarP(&scanInfo.Submit, "submit", "", false, "Submit the scan results to Kubescape SaaS where you can see the results in a user-friendly UI, choose your preferred compliance framework, check risk results history and trends, manage exceptions, get remediation recommendations and much more. By default the results are not submitted")
scanCmd.PersistentFlags().BoolVarP(&scanInfo.OmitRawResources, "omit-raw-resources", "", false, "Omit raw resources from the output. By default the raw resources are included in the output")
scanCmd.PersistentFlags().BoolVarP(&scanInfo.PrintAttackTree, "print-attack-tree", "", false, "Print attack tree")
scanCmd.PersistentFlags().BoolVarP(&scanInfo.ScanImages, "scan-images", "", false, "Scan resources images")
scanCmd.PersistentFlags().BoolVarP(&scanInfo.IsSecurityView, "security-view", "", false, "Show security view")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use --view instead

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would this work when using security view with resource view?


scanCmd.PersistentFlags().MarkDeprecated("silent", "use '--logger' flag instead. Flag will be removed at 1.May.2022")
scanCmd.PersistentFlags().MarkDeprecated("fail-threshold", "use '--compliance-threshold' flag instead. Flag will be removed at 1.Dec.2023")
Expand Down Expand Up @@ -117,9 +127,38 @@ func GetScanCommand(ks meta.IKubescape) *cobra.Command {

scanCmd.AddCommand(getControlCmd(ks, &scanInfo))
scanCmd.AddCommand(getFrameworkCmd(ks, &scanInfo))
scanCmd.AddCommand(getWorkloadCmd(ks, &scanInfo))

isi := &imageScanInfo{}
scanCmd.AddCommand(getImageCmd(ks, &scanInfo, isi))

return scanCmd
}

func setSecurityViewScanInfo(args []string, scanInfo *cautils.ScanInfo) {
if len(args) > 0 {
scanInfo.SetScanType(cautils.ScanTypeRepo)
scanInfo.InputPatterns = args
} else {
scanInfo.SetScanType(cautils.ScanTypeCluster)
}
scanInfo.SetPolicyIdentifiers([]string{"clusterscan", "mitre", "nsa"}, v1.KindFramework)
}

func securityScan(scanInfo cautils.ScanInfo, ks meta.IKubescape) error {

ctx := context.TODO()

results, err := ks.Scan(ctx, &scanInfo)
if err != nil {
return err
}

if err = results.HandleResults(ctx); err != nil {
return err
}

enforceSeverityThresholds(results.GetData().Report.SummaryDetails.GetResourcesSeverityCounters(), &scanInfo, terminateOnExceedingSeverity)

return nil
}