Skip to content

Smoke tests and samples #1193

@dlion

Description

@dlion

Summary

I noticed that some of our samples don't work properly with the current version of lifecycle. (i.e. buildpacks/samples#167, buildpacks/samples#163)

Those samples can be very useful for identifying possible regressions and for the community, we use them in our documentation, we often use them during our troubleshooting and we should treat them as a first-class citizens.

Manual solution

In order to verify that our code didn't have any regression and to maintain them up to date being sure that they work every time we should try them every new release of a stable version of pack/lifecycle.

Downside

It requires lots of time and effort from our side and we can potentially forget about it.


Proposal

So I thought about an automatic and more efficient way to do so:

My proposal would be to have some smoke tests that we run (or that our CI runs -better-) every time we release a new version of lifecycle/pack towards those examples.
It will force us to update those samples with the latest changes in order to release a stable version and of course to verify that we didn't introduce any regression.

The process can looks like:

  1. Build lifecycle
  2. Run lifecycle components one by one using our samples
  3. Do that for every important/basic scenario
  4. Check for failures
  5. If every thing went well we can release otherwise we need to adjust our samples first.

Context

https://www.guru99.com/smoke-testing.html
https://martinfowler.com/articles/practical-test-pyramid.html

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions