Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions modules/osdk-scorecard-store-output.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
// Module included in the following assemblies:
//
// * operators/operator_sdk/osdk-scorecard.adoc

:osdk_ver: v1.16.0

:_content-type: PROCEDURE
[id="osdk-scorecard-store-output_{context}"]
= Storing scorecard test results

You can enable local storage of scorecard test results by using the `scorecard` command or by modifying your `config.yaml` file.

[IMPORTANT]
====
There is a known issue with the Operator SDK scorecard storage utility. Currently, running the `operator-sdk scorecard` command with the `--test-output` flag returns an empty directory. There is no known workaround at this time. (link:https://bugzilla.redhat.com/show_bug.cgi?id=2017669[*BZ#2017669*])
====

.Procedure

. If you want to run a scorecard test from the command line and store the results, run the following command in a terminal window:
+
[source,terminal]
----
$ operator-sdk scorecard <bundle_directory_or_image> --test-output=/<output_directory> <.>
----
<.> Define the output directory.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, I'd rewrite this whole thing without the broken flag and be like "this command store the test results in the test-output directory of your working directory." When the bug is fixed, update the docs to describe the flag.

Copy link
Contributor

@sferich888 sferich888 Mar 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 The way this is written makes this read as if 'we have this cool new feature, but none of it works - good luck'.

I wouldn't document this till the bug was/is fixed; only document the way's in which it properly works.

+
If you do not specify a directory, the command stores the test results in the `test-output` directory of your working directory.

. If you want to enable local storage in your configuration file, add the following to your `config.yaml` file:
+
[source,terminal]
----
tests:
- entrypoint:
- scorecard-test
- <test_entrypoint> <.>
image: quay.io/operator-framework/scorecard-test:v1.10.0
labels:
suite: <test_suite> <.>
test: <test_name> <.>
storage:
spec:
mountPath:
path: /<output_directory> <.>
----
<.> Replace `<test_entrypoint>` with the command and arguments that you want to run, such as `scorecared-test`.
<.> Define the test suite that you want to run, such as `olm`.
<.> Define the test name that you want to run, such as `olm-bundle-validation-test`.
<.> Set the the location of directory to store your test results, such as `test-results`.

:!osdk_ver:
1 change: 1 addition & 0 deletions operators/operator_sdk/osdk-scorecard.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ include::modules/osdk-scorecard-config.adoc[leveloffset=+1]
include::modules/osdk-scorecard-tests.adoc[leveloffset=+1]
include::modules/osdk-scorecard-run.adoc[leveloffset=+1]
include::modules/osdk-scorecard-output.adoc[leveloffset=+1]
include::modules/osdk-scorecard-store-output.adoc[leveloffset=+2]
include::modules/osdk-scorecard-select-tests.adoc[leveloffset=+1]
include::modules/osdk-scorecard-parallel.adoc[leveloffset=+1]
include::modules/osdk-scorecard-custom-tests.adoc[leveloffset=+1]