Golden & roundtrips on *all* Api types

[Anviking]

Issue Number

#91

Overview

• Added roundtripAndGoldenPerType + machinery to run tests on all Api Capture- and return-types.
• TODO: Go through the servant combinators. ReqBody?
• WIP machinery for taking apart ADTs and running roundtripAndGolden on each component.
  • It was able to run roundtripAndGolden for the type of each field of a record
  • I was unable to hook it up with the Api discovery
  • I didn't figure out how to "catch" an hspec failure, such that we could then run more detailed tests on the components on the failing type (see comment for GRecursiveCheckJSON)

Comments

[KtorZ]

Looks good modulo the case for query / path parameters within Capture.
It'd be nice one day to also get the roundtrip automagically for nested types, yet, it's not trivial 🤔

[KtorZ]

That one is incorrect. There's no reason for a param t to have a a JSON instance. Instead, we should be checking for HttpApiData roundtrips here!

[Anviking]

It'd be nice one day to also get the roundtrip automagically for nested types, yet

@KtorZ I just discovered that

hspec-golden-aeson contains

goldenADTSpecs :: forall a. (ToADTArbitrary a, Eq a, Show a, ToJSON a, FromJSON a) => Settings -> Proxy a -> Spec

I'm checking how it works. If it is close to trivial we could add now, otherwise later. Probably in a separate pr regardless.

Did not get this working at all 🤷‍♂️ (as in simple to set-up, but it only ran tests on the original type for some reason)

[KtorZ]

@Anviking

-> TODO: Go through the servant combinators. ReqBody?

No need. We'll add them as we add API types.

[Anviking]

Closing for now, the value wasn't clear enough. A few problems:

Lacking reduction of test cases

The pr had logic for generating test-cases for a instead of [a], but this was not generalised to other types than List.

Not testing ADT components

-- We are relying heavily on generically derived Aeson instances. This is used
-- to trace the failure of @roundtripAndGolden@.
--
-- Example:
-- @
--    data Record { a :: A, b :: B, c :: C, ds :: [D] }
--        deriving (Generic, ToJSON, FromJSON)
-- @
--
-- If we hand-write faulty Aeson instances for 'D', we'd like to discover that
-- easily when checking roundtrips and goldens on 'Record'.

Dangers of regeneration

If we believe we're making a purely additively change to a type, but screw something up, the tests won't help us. They are re-generated. (actually, no, but not re-generating them is also a problem)

For the future

• In its current form this ensure that we haven't missed any roundtrip or golden tests in the Api-type. It serves like a backup, but shouldn't replace manually listed test cases.

• With some more careful thought, we might be able to do something similar — but valuable —in the future.