[ITAS] As a tester, I want to easily be able to create API responses from the API doc so that I don't have to craft fake response myself

[maxi297]

Problem

Creating responses for the HTTP mock to return is cumbersome because most of the content of the response is not relevant to the tests and having this big blob of text in the test hinders readability. Having those in separate files is not that much better because you either end up with one file per test to manage (imagine if the response changes!!) or the specifics of the content of the file is not readable from the test. Hence, we want a expose only what is relevant to a test without having too much maintenance.

Proposed solution

Have an interface that only expose the important things. Domain wise for sources, it means where the records are in the response. For stripe specifically, there are other things:

• how pagination is enabled
• what are the ids of the records (this is important for pagination as well)

Hence, proposed interface is:

class RecordBuilder:
    def with_id(self, identifier: Any) -> "RecordBuilder":
        pass

    def with_cursor(self, cursor_value: Any) -> "RecordBuilder":
        pass

    def build(self) -> Dict[str, Any]:
        pass


class ResponseBuilder:
    def with_record(self, record: RecordBuilder) -> "ResponseBuilder":
        pass

    def with_pagination(self) -> "ResponseBuilder":
        pass

    def build(self) -> Dict[str, Any]:
        pass

In order to create those builders, the user should only have to

• copy/paste a response (either from the API doc or from the actual server if our dataset allows it) in a file. Note that the response should have one record
• provide the path to the records
• provide the path of the id within the record
• provide to logic with which the API defines is there are more pages (only if this logic is not based on the number of records)

It is fine to take a couple assumptions like "the files will always be in the same folder" (in the PoC, I used airbyte-integrations/connectors/source-<source name>/unit_tests/http/responses/<resource>.json)

Note on pagination

We could decide to be more generic by using with_field("key", "value"). It would probably also be more explicit in the tests as it would show exactly which field is impacted. However,

• It is more flaky as if pagination change for an API, all those with_field will need to be updated (which should be almost all covered with a search and replace so I'm not support worried about this)
• It allows for the devs to do more shenanigans

Acceptance Criteria

• Code in the CDK with tests but not used yet (will be used in [ITAS] Integration tests for all empty streams in source-stripe #32871)