Skip to content

@heptibot heptibot released this Sep 30, 2019 · 26 commits to master since this release

Sonobuoy v0.16.1

Sonobuoy v0.16.1 mitigates a major bug found in the upstream E2E conformance image. Users are highly encouraged to update. See #910 for more details.

What's new

Golang-based test runner for Kubernetes 1.16.0+ clusters.

Within the test image used for end-to-end tests are some wrappers which invoke those tests. Historically those were written in bash but various problems have occurred which caused us to add a Golang-based runner. The benefits of this are:

  • avoids a major problem where 1.16.0 clusters with the bash runner would not report test results properly when there are failures
  • improved error handling and logging
  • ability to further customize the test run by allowing arbitrary flags to be passed via the env vars. This means that many more testing options are supported including provider specific settings that were previously unsettable.

What's better

  • sonobuoy results will print results for all plugins by default. This way you don't have to remember which plugins were run or how their names are exactly spelled to figure out what is inside a tarball.
  • sonobuoy gen plugin added a --format flag which allows you to specify the result format.

Changelog

c9c2a46 Prepare for v0.16.1 Release (#921)
2c92062 Tweak readme with info about results. (#917)
0e16b5e Default to using the golang-based conformance runner (#916)
bf2fa52 Add a small test which runs a basic plugin (#903)
06f638e Allow the --plugin flag to specify a directory of plugins (#911)
d9ca4ba Prints all plugins when invoking sonobuoy results by default (#909)
aae48ef Adds blog post summarizing improvements made up until 0.16.0 (#908)
6e5dcaa Adds a testImage for our integration tests (#907)
d42d0f1 Adds the results format as a flag parameter for gen plugin (#904)
e30312b Add kind and dep to path in Travis build (#902)
7473b3d Remove unused dependency from Gopkg.toml (#898)
a8e4c1c Add a test to check that version fields are set (#894)

Assets 6

@heptibot heptibot released this Sep 19, 2019 · 38 commits to master since this release

Sonobuoy v0.16.0

This release corresponds to the Kubernetes v1.16.0 release. It takes place very close to our previous release, Sonobuoy v0.15.4, and does not contain any user-facing changes other than supporting Kubernetes v1.16.x.

Changelog

6db0b0b Prepare v0.16.0 release (#896)
b0fb524 Rename stress test directory to match package name (#893)
0b77329 Fix script to make new versioned docs (#892)

Assets 6

@heptibot heptibot released this Sep 17, 2019 · 41 commits to master since this release

Sonobuoy v0.15.4

What's new

  • Supported modes

The supported "modes" for sonobuoy run have changed. The supported values are now quick, non-disruptive-conformance, and certified-conformance. This is due to Kubernetes changes in v1.16 which will be allowing disruptive tests to be included in Conformance runs. Sonobuoy, by default, will seek to protect your workloads by avoiding those disruptive tests unless you explicitly state that you want to run in certified-conformance mode.

To be clear, sonobuoy run is most appropriate as a safe way to test that your cluster is in a good state.
sonobuoy run --mode certified-conformance will run all the conformance tests for submission to the CNCF Certified Kubernetes program.

  • Progress updates

Sonobuoy now has support for collecting and reporting on the progress of plugins while they are still running. This functionality is currently just in Sonobuoy itself and is not yet implemented in the upstream E2E tests, where it is most useful. A new [example][https://github.com/heptio/sonobuoy/tree/master/examples/plugins/progress-reporter] has been added to illustrate how you can utilize this functionality in your own plugins.

What's better

  • In v1.16 Kubernetes has a few E2E tests which may evict user workloads from a node (see the above item about certified-conformance mode. To ensure that Sonobuoy and your plugins run appropriately and do not get evicted, a new toleration is added by default.
  • When tailing logs via sonobuoy logs -f, if Sonobuoy sees a container but it is not yet running, we will continue to watch the container and start streaming logs once it enters the running state.
  • If a plugin fails to report any results and is considered an unknown result, that will bubble up and cause the plugin to properly be marked as unknown rather than passed by default. This reduces the number of false positives possible.

Changelog

3d6c1be Bump version and add v0.15.4 docs (#886)
90a586f Add certified conformance mode and non-disruptive-conformance mode (#885)
cb43587 Progress reports for plugins (#869)
8bd715b Remove 'extended' mode (#884)
9eae93e Minor change to retrigger blog publishing (#882)
06fcf66 Add blog post for Customizing PodSpecs (#879)
f7e2366 Add a disruptive taint to list of default tolerations (#878)
0f9e963 check for container running before following logs (#706)
e03eebd Remove unnecessary legacy branding (#873)
cfde6a0 Update to Go 1.13 (#871)
3975a9e Show logs for a specific plugin (#868)
bc64713 Adjust MASTER_URL to be just the hostname (#867)
a603fdb Bubble up unknown results just like failures (#866)
bfa3ade Fix broken links and reference to conformance image (#864)

Assets 6

@heptibot heptibot released this Aug 30, 2019 · 55 commits to master since this release

Sonobuoy v0.15.3

What's new

  • The results post-processing for JUnit now supports output which contains a testsuites object consisting of multiple test suites rather than just being able to support a single testsuite object.

What's better

  • Plugin definitions (e.g. output from sonobuoy gen plugin and input for the --plugin flag) no longer have a result-type field. This was usually identical to the plugin-name field and only led to confusing users or causing issues if those values were not identical.

Changelog

4737048 Update version and add new docs for v0.15.3 (#862)
250dfc1 Update release docs for new build flow (#861)
3b14025 Remove plugin.ResultType due to confusion/lack of need (#857)
2d47854 Add documentation for custom PodSpecs (#859)
a268401 Add junit support for non-e2e tests/multi-suite (#851)
451b313 Use MarshalIndent on json goldenfiles (#856)
973e782 Rename master to aggregator (#847)
58b331c Update gems (#850)

Assets 6

@heptibot heptibot released this Aug 16, 2019 · 63 commits to master since this release

Sonobuoy v0.15.2

What's new

  • PodSpec fields can now be specified for individual plugins.
    • In addition to specifying the container and extra volumes required for a plugin, you can now modify the PodSpec used by Sonobuoy to launch the plugin. You can view the default PodSpec used by Sonobuoy for both Job and DaemonSet plugins by using the new flag --show-default-podspec on both the gen and gen plugin commands. This will allow you to modify the default one, or use it as a basis to provide your own for existing plugins. Sonobuoy will still add resources that it requires to this PodSpec before running the plugin (such as containers to run the plugin image and the Sonobuoy worker) but won't remove or change any settings you provide.
  • Plugin results are now placed onto the aggregators status annotation
    • This means you can now see if plugin had any test failures without even downloading the results tarball. You can use sonobuoy status to see the top-level results (e.g. passed/failed). Alternatively, a new --json flag was also added so you can see/parse the full status object.
    • We also added some metadata to the status object so you can relate it to the tarball you download. Fields include filename, sha256, and size.

What's better

  • OwnerReference set on plugin pods/daemonsets
    • By setting this metadata field when the aggregator launches plugins, we now ensure that plugin resources are deleted if the aggregator pod is deleted.
  • The sonobuoy results command now reports unknown on junit plugins if no xml files were processed instead of passed. This helps properly identify cases where a plugin may not generate the junit properly.
  • Added Details field to the results.Item object. This means that sonobuoy results --mode=detailed can not just list the tests which failed but also the failure message and captured stdout.

Bug fixes

  • Fixed a bug which caused Sonobuoy to not filter based on namespace properly when running queries after plugins are run.

Changelog

d776707 Bump version of Sonobuoy and add new docs for v0.15.2 (#846)
031d29d Set OwnerReference on Plugin resources (#845)
2bec2b1 Modify some old docs to avoid liquid syntax warnings (#844)
2fb844c Ensure we filter resources by namespace (#842)
c7d0673 Ensure a plugin isn't reported as passed if no results were read (#841)
5238cfa Add Details map to post-processed Items (#838)
0fb9865 Add ability to override pod spec options (#837)
1cce037 Add results info to the status annotation and status cmd output (#829)
3a04f8f Use gcloud auth instead of docker-credential-gcr (#836)
83e39f2 Add image building/pushing back to Travis (#835)
4c07889 Remove use of templates for plugin Jobs and DaemonSets (#834)
94702fb Sync master/root README.md and provide a script to do this (#831)
f68cb44 Remove plugin command (#833)
01a2fcb Fix SEO title output on docs
4cda024 Add SEO functionality to the site
838ea5a Add 0.15.1 blog post (#828)

Assets 6

@heptibot heptibot released this Aug 2, 2019 · 81 commits to master since this release

Sonobuoy v0.15.1

What's new

  • New command sonobuoy results to access results from any plugin.
    • See [https://sonobuoy.io/docs/v0.15.1/results/][the docs] for more details on this new command. TL;DR; it is now possible to read results from any plugin without opening up the tarball.
  • New flag --wait-output
    • The flag defaults to 'silent' (existing behavior) but you can also adjust it to 'spinner' which will ensure that CI systems (and users) don't think that the command has hung.
  • New plugin definition option SkipCleanup
    • This feature increases your ability to run tests and experiments which may need more manual debugging after the test run. Previously, Sonobuoy would automatically delete all the plugin pods once they were completed preventing manual inspection.

What's better

  • Kubernetes version support
    • If the Kubernetes version of the cluster is greater than the maximum version supported by Sonobuoy, a warning is displayed instead of an error. This should make it easier for users who are trying to test on new clusters when Kubernetes versions get released.
  • The plugin definitions are written into the 'plugins/' directory for higher visibility.
  • Multiple commands now rely on labels instead of exact pod names when targeting the Sonobuoy aggregator pod. This helps facilitate situations where Sonobuoy is being deployed as a Job, Deployment, or ReplicaSet.
  • Extra query options
    • Numerous options were added to facilitate getting pod logs to be more in line with upstream options.
  • Return exit code 1 if sonobuoy can't pull all images required for the e2e tests

Bug fixes

  • Fixed a bug where the UUID may not be set on a config in certain code paths.

Changelog

4acae8c Bump version to 0.15.1 and add versioned docs
b89b8bb Add some documentation for new results command (#825)
7713c02 Allow multiple result files to be post-processed (#824)
e39cde3 Default to ResultFormat=raw if unset (#822)
d91bc40 Save results in a canonical format via postprocessing (#813)
a6dc3a3 Update some vars and literals to use consts (#816)
18d5aea Change max supported version error to warning (#815)
d739225 Merge pull request #622 from ekj1711/wait-emit-output
fe23f98 Write plugin definitions in their respective folders (#812)
f0ffecc Remove use of global sonobuoy pod name (#811)
020a7e4 Remove plugin.Definition (#810)
5519e20 Wait should emit some output
6240935 Preserve single file result names and keep job/ds depth the same (#785)
c942509 Ensure only failed tests are run with --rerun-failed (#801)
e3e7514 Pin images used in kind config and reduce number of workers (#803)
160b58a Upgrade kind to v0.4.0 (#804)
4a5c1fd Create a UUID for loaded configs if necessary (#807)
0152d01 Add Docker to requirements for sonobuoy images (#805)
700ac79 Merge pull request #800 from MengmengZHANG/fieldSelector
8380685 add fieldSelectors & labelSelectors for getPodLogs (#778)
5009540 Generate Plugins ConfigMap if no plugins specified (#799)
443f20a Fix a broken link on "Conformance Testing" page (#798)
90cd0d1 Allow cleanup to be skipped on a per plugin basis (#796)
ad75ce4 Merge pull request #795 from jonasrosland/redirect
4e5187e Small CTA fix
456ec57 Add redirect for docs
afcd78a Fix links to doc pages on new site (#794)
c029d9f Fix preflight test error message (#791)
dca7c17 Fix retrieve command when using dynamic pod name (#790)
bc9e84b Look up sonobuoy master pod name by label (#787)
f298701 issue 777 add more query options for getPodLogs (#788)
c3b5afa Return exit code 1 if sonobuoy can't pull all images required for the e2e tests (#786)
fd7722d Add e2e-repo-config flag to images pull command (#781)
c7c4345 Add new blog post for dry-run and --plugin-env flag (#774)
2439cd4 Fix logo reference on main README (#773)
20e0df5 Update README to point to docs site (#771)
6811a38 Add v0.15.0 doc snapshot and automate process (#770)

Assets 6

@heptibot heptibot released this Jun 24, 2019 · 117 commits to master since this release

Sonobuoy v0.15.0

What's new:

Versioned Documentation

The Sonobuoy website now has documentation available for each release of Sonobuoy going back to v0.13.0. As new versions are released, documentation specific to that version will be available.

The docs for this release are available at sonobuoy.io/docs/v0.15.0.

What's better:

This release takes place very close to our previous release, v0.14.3, and focuses more on bug fixes and improving the contributor experience.

Bug fixes

  • Mitigated a bug where there would be a failure due to the container being terminated even though results had been sent. Sonobuoy will now wait longer for results after they have been sent and the container terminated before deeming the run to be a failure.
  • Fixed a bug where a panic would happen if a nil configuration parameter was passed to most of the functions in the Sonobuoy client library.

Contributor experience improvements

  • Improved the debugging experience in the CI builds. More logging has been added to help track down and debug issues when builds fail.
  • Added a check to ensure that a developer commits any changes resulting from dep changes.
  • Improved tests which previously checked that the build version was correctly included when generating a Sonobuoy config. These tests needed to be updated on every release but no longer require this as they instead check a static version.
  • Improved the documentation for the Sonobuoy release process to include more details on steps required for minor version increases which correspond to new Kubernetes releases.
  • Added a native Makefile target so that Sonobuoy can be built locally using the same flags as in CI without using Docker.

Changelog

7ad3675 Prepare v0.15.0 release (#769)
c34d3ee Update release process documentation (#768)
41ce5d4 Tweak the timing of considering pod as failing to be more permissive (#764)
ee916a1 Add basic validation of configs (#763)
f1fbf8a Add versioned docs to Sonobuoy site (#756)
7e5103d Fixup debug line to target regular files and print file name (#762)
e257b50 Apply changes from running gofmt (#758)
daa7ab3 Add Bridget to the team on the website (#760)
6c522c8 Add native Makefile target (#757)
8ac5ca2 Ensure dep has been run in CI (#755)
985c071 Update golden file tests to use static versions (#751)
2630934 Fixup to travis debug lines (#753)
4901e55 Add vscode to gitignore (#749)
7d7bd81 Set GitSHA when using goreleaser (#750)

Assets 6

@heptibot heptibot released this Jun 14, 2019 · 131 commits to master since this release

Sonobuoy v0.14.3

What's New:

A new Sonobuoy website: sonobuoy.io

The website is generated from documentation in the repo itself and uses Netlify to generate a site preview when we change those documents.

Setting arbitrary env vars on plugins

A new flag --plugin-env can be used to set any env var on any plugin. This will prevent you from having to (1) add a flag to Sonobuoy to edit your env vars (2) save and edit the YAML for the plugin in order to modify the env var.

You can set env vars on any plugin (whether or not it is a custom or built-in plugin) by referring to the plugin by name:

sonobuoy run --plugin-env e2e.E2E_DRYRUN=true --plugin ./myplugin.yaml --plugin-env myPlugin.FOO=BAR

Custom annotations on Pods

In some situations you may need to have the Sonobuoy pods all lauched with custom annotations, for example, to use KIAM permissions. A new field, CustomAnnotations, in the config file format supports this.

For example, to add the annotation foo:bar to your pods:

sonobuoy gen config | jq '.CustomAnnotations={"foo":"bar"}' > myConfig.json
sonobuoy run --config myConfig.json

What's Better:

Improved the feedback loop when a plugin failed to run

Up to this point, if a plugin failed to run, the logic in the server overreacted and would exit entirely. Now, it reports the error for that plugin and continues the run as normal.

Improved the feedback loop when a plugin exited without returning results.

In this case, the Sonobuoy sidecar would continue to wait for the results from the main container (your plugin) despite it already having exited. Now, the aggregator will watch for this condition and report it as an error running the plugin. This avoids the aggregator server hanging until the timeout waiting for results.

Improved stability in situations where network failures are high.

The Sonobuoy sidecar on the plugins automatically was retrying to send plugin results if the connection failed for some reason. However, the aggregator server was not accepting retries appropriately. This led to situations where your plugin results would be missing or truncated, despite the pod logs of the plugin showing more full results.

Other bug fixes:

  • Fixed a bug which could cause the Sonobuoy status to never report completed if no plugins were run.

  • Fixed a bug which prevented the user from providing an empty array of resources to gather (indicating to gather them all).

  • Fixed a bug which caused numerous resource directories to be created in the results tarball, even if no resources of that type existed.

  • Fixed a deadlock issue which could occur if you tried to run sonobuoy logs when pods were not yet ready.

  • Fixed a bug which would have caused a manually specified conformance image to be improperly manipulated causing a failed image lookup.

  • Documentation improvements:

    • Clarified the support matrix (3 minor versions total)
    • Added documentation describing how to use a Sonobuoy image which is in a private registry.
    • Fixed a broken link in the conformance docs
  • Code cleanup:

    • Combined two code flows into one for reuse when handling plugin results.
    • Made GetStatus library command have a signature consistent with other methods
    • Code cleanup including removing redundant and confusing fields from GenConfig

Changelog

6d02613 Fix build and cleanup test to ensure clean state for release (#747)
439de5f Bump version to v0.14.3 (#743)
6466ac3 Fix excerpt separator being shown in blog posts (#742)
3324bc4 Add checks for if a container has terminated (#734)
a7bba9b Add daemonSet monitoring tests (#740)
61ac7be Add tests for job.Monitor logic (#739)
edb4bf5 Adjust README description of support matrix (#737)
4d4ff7a Merge pull request #730 from johnSchnake/retryResults
f51d88d Reprocess results if there was a processing error
98d7d32 Unify http/non-http result handling (#729)
cfd9cfc Use a struct for GetStatus to be consistent (#731)
a07aed2 Add ability to specify custom annotations (#723)
f77a792 Documentation usability and readability update (#727)
d5c624c Fix/simplify the config loading logic enabling empty resources (#725)
f2c885d Add initial annotation to the pod when starting (#724)
d34e9cf Don't create resource directories if not querying for resources (#726)
e9aba66 Properly report error when a plugin fails to run (#720)
90b3df8 Ensure plugin selection list and resources can be empty (#722)
0fa9c4f Merge pull request #712 from smalltalk-ai/master
34a4918 Update version number
dad5f21 Update README | Remove enhancements
875aae4 Update docs from master branch not gh-pages
702c22a Updates to homepage links
68df46a Adding in Jekyll site
9f0c2b0 Set any plugin env vars from the CLI (#709)
9a19511 Fix race/deadlock issues with logs when a pod is not ready (#719)
affd4fb Ensure conformance image version is prefixed with 'v' (#717)
770b85b Fix broken link in conformance doc (#718)
c76f2f5 Use go modules when getting KinD (#714)
a4e9a32 Merge pull request #705 from johnSchnake/pullSecretsDoc
65e2319 Merge pull request #704 from padlar/padlar/issue-691
108d168 remove redundant fields from GenConfig
9070e29 Add documentation about how to pull private Sonobuoy image
3bef2a4 Add example for command runner plugin

Assets 6

@heptibot heptibot released this Apr 29, 2019 · 165 commits to master since this release

What's New in v0.14.2

Improved Plugin Support

The major improvement on this release are the --plugin flag and the sonobuoy gen plugin command.

Use sonobuoy gen plugin to generate a custom plugin definition and then use --plugin with either sonobuoy gen or sonobuoy run to select which plugins to run. For example, to run a custom plugin and the e2e tests together, run the command :

$ sonobuoy gen plugin --name myPlugin --image myImage > customPlugin.yaml
$ sonobuoy run --plugin customPlugin.yaml --e2e

See more details in a this new example.

In addition, the aggregation server will, by default, run all the plugins it can find when it starts. This differs from previous releases where it would only run plugins specified in the Sonobuoy configuration's Plugins field. Now, by default that field is null and it will run everything. If you want, you can still specify that field manually like you did before to require a specific list of plugins to run. If you want to skip all plugins you can provide the empty array ([]).

Dynamic Client for Queries

The query logic (which gathers information about your server after the plugins run) used to have a discrete list of resources it could query. We've updated this so that it dynamically crawls the API to enable it to gather more information. If you want to limit the values it gathers, just set the Resources field in the Sonobuoy configuration file to list the API resource names you want. By default it still explicitly lists the resources it intends to gather, but if you remove the Resources field you can see it gathers everything it can (with the exception of secrets). This can be really useful if you want to see things like CRDs.

The logs also now show explicitly which items the query logic is skipping; that way you can be confident about what you are querying for and what items you could add.

Sonobuoy Configuration File Generation

We added a new command: sonobuoy gen config which will spit out the default Sonobuoy configuration. This has two purposes:

  • Helps you understand what the default values are
  • Makes it easy to take the default and modify a single value or two which are of interest. Now you don't have to wonder what the fields are named or what the defaults are if you just want to slightly tweak something.

ImagePullSecrets for Sonobuoy Image

If you are hosting the Sonobuoy image in a private Docker registry which required a password, in previous releases it was not possible to get Sonobuoy to work without a custom build because the templates and data types didn't "know" the imagePullSecrets field.

Support for that has been added. It requires setting the ImagePullSecrets field in the Sonobuoy configuration and then also manually adding the YAML for your secret in the output from sonobuoy gen and running it all manually via kubectl apply. Future releases may consider how to better handle creating additional resources like this.

Server Timeout Flag

One of the most commonly edited fields in the Sonobuoy config file is the timeout for the aggregation server (how long it will wait for plugins to complete). We added a new --timeout flag to sonobuoy gen and sonobuoy run to make this easier.

Context Flag for KUBECONFIG

A new --context flag has been added everywhere --kubeconfig was supported in order to automatically handle multi-context config files.

Other changes

Preflight checks will now error if you try to run Sonobuoy in an existing namespace. This is to help prevent the situation where you create Sonobuoy in a namespace which you don't want to later destroy (meaning you've got to clean things up manually). If you really want to create Sonobuoy in an existing namespace, you still can: just provide the --skip-preflight flag.

Changelog

ebc7916 Add coverage profile to gitignore to ensure clean state after tests
a38e39e Bump version to v0.14.2
d5927ae Fix servergroups query
ee6ecc8 Update queryies to use the dynamic client
a85b38f Add ability to set args when defining a plugin
ec8b6c6 Properly set default for plugin job driver
bf155c8 Support imagePullSecrets for sonobuoy workers
aaa5e13 Add CLI flag for aggregation timeout
1b56c31 Add plugin flag for custom, local plugins
1a72321 Add sonobuoy gen config file and documentation
012648c Tighten preflight check to err on existing namespace
a5d8610 Update client go and other deps
7af1d5a Run all loaded plugins by default
89bb5b4 Modify the json serialization of config object
f488867 Add test that will run through command creation
b420055 Fix flaky unit test
8f123df Change logic in template to use a list of plugin manifests
0f8ab46 Add gen plugin command to generate boilerplate for plugins
ee335d2 Centralize the process of going from kubeconfig to sonobuoy client
d54a882 Defer assignment of Docker gcr.io secret
566e80a Add context flag whenever kubeconfig flag is added
8eb9fb5 Add CodeCov reports

Assets 6

@heptibot heptibot released this Apr 9, 2019 · 190 commits to master since this release

Changelog

797da23 Bump version for 0.14.1
fc4acbb Use heptio kube-conformance image for any cluster < v1.14.1
33f61c2 Fix CRD query
c3a2050 Error e2e command if no e2e results
18a07b9 Add validation to focus/skip regexp flags
755fafb Update release documentation
a027811 Add additional documentation about testing in airgap

Assets 6
You can’t perform that action at this time.