Skip to content
This repository has been archived by the owner on Mar 1, 2019. It is now read-only.

(CI) Have a way to generate and verify exitence of golden tests #124

Open
KtorZ opened this issue Nov 29, 2018 · 1 comment
Open

(CI) Have a way to generate and verify exitence of golden tests #124

KtorZ opened this issue Nov 29, 2018 · 1 comment
Labels
ACCEPTED ADR has been accepted

Comments

@KtorZ
Copy link
Contributor

KtorZ commented Nov 29, 2018

Context

API breaking changes are bad and also tricky to identify somehow. Right now, there's nothing really helping us not breaking the API apart from our own discipline. Despite being disciplined, something can also slip through our attention and get merged, unnoticed. A simple step towards this would be to add a few golden tests to make sure that API types aren't actually changing in a non backward-compatible ways.

If a given API type has one or a few associated golden test, hopefully, upon making a change, the underlying change would fail. Then, we can assess whether this change is a breaking change or just a soft extension.

Decision

We want to write such golden tests for existing API types. But because tracking the existence of those tests can be tedious, we'd also like ideally, to have a way to enforce the existence of such tests in such way that the code wouldn't even compile without us specifying them.
Since the API is completely defined in terms of a single type by the mean of Servant combinators, we can work out some checks using a generic approach.

Acceptance Criteria

    • Each API types that is exposed must have one or more golden tests for its corresponding JSON or HttpApiData instances.
    • The existence of such tests should be enforced for existing and future types automatically.

PR

Number Base
#? develop

QA

Criteria Coverage
? -

Retrospective

@KtorZ KtorZ self-assigned this Nov 29, 2018
@KtorZ KtorZ changed the title TODO: Have a way to generate and verify exitence of golden tests Have a way to generate and verify exitence of golden tests Dec 10, 2018
@KtorZ KtorZ added the PROPOSED ADR just proposed label Dec 10, 2018
@KtorZ KtorZ changed the title Have a way to generate and verify exitence of golden tests (CI) Have a way to generate and verify exitence of golden tests Dec 10, 2018
@KtorZ KtorZ added ACCEPTED ADR has been accepted and removed PROPOSED ADR just proposed labels Dec 13, 2018
@Anviking
Copy link
Member

Anviking commented Dec 13, 2018

We could also consider generating golden tests and writing them to disk automatically so that API changes are tracked in git. (This would likely require the modifications to the API being translated very well, and deterministically into changes to the golden tests though.)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
ACCEPTED ADR has been accepted
Projects
None yet
Development

No branches or pull requests

2 participants