[ITAS] As a tester, I want to safely mock HTTP requests so that I don't have to rely on the actual API

[maxi297]

In order to only validate the integrations between the component from the entrypoint to the HTTP request, we need to mock the actual HTTP request that would be performed. We could directly use a lib like resquests-mock but this generates some issues:

• If the library doesn't support something we want, we will have to bleed code everywhere
• If the library is updated with a breaking change, this breaking change might affect all the tests individually

Hence, it makes sense to wrap this with an interface that makes sense to connector testing and that will evolve at the same pace as we will like. For the scope of this issue, here are the queries we would like to support:

• mocking get requests
  • match requests on URL (including query params) and on the headers that are provided (for example, if the matcher has header1: 1 and header1: 1 and header2: 2 are provided, match should be successful but if the matcher has header1: 1 and header1: 2 then the match should fail). Error messages should be clear in all cases (one case to look more into is the header matching case as requests-mock does not explicit the headers when there is no match)
  • return the response provided on match
• multiple get requests can be configured at the same time
• if not all the requests are matched during the scope of a test, the test should fail

Those capabilities need to be available for all the sources. The proposed solution for this is to add a test package in the CDK without any safeguard for devs not to import this in production code. If a safeguard can be implemented easily, go for it but it is not a blocker.

Questions:

• Are we fine using requests-mock which would only work for sources using requests to perform HTTP requests?

Acceptance Criteria

• Code in the CDK with tests but not used yet (will be used in [ITAS] Integration tests for all empty streams in source-stripe #32871)

[maxi297]

grooming note:

• We should support responses that are errors. Hence the framework should allow to specify an HTTP status code
• For now, we don't need to count the number of matches for each matcher but eventually we might want to use that to see if caching works for example
• For now, we would support only one response per query but eventually, we would like to have multiple responses. Example is salesforce where we poke an endpoint until the report is fully generated in which case the response changes