Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix td operators tests by using HTTPS #1553

Merged
merged 7 commits into from Apr 9, 2021
Merged

Conversation

komamitsu
Copy link
Contributor

@komamitsu komamitsu commented Apr 7, 2021

Treasure Data's API endpoint doesn't accept HTTP requests now. In this PR, we make all accesses to TD via HTTPS in the test code.

But we need to tentatively skip some test verifications (e.g. acceptance.td.TdIT#testRetries()) in addition.

In this kind of tests, td operator accesses to td-api through a HTTP proxy server org.littleshot.proxy created in the test code for these purposes:

  • Make some of requests to td-api fail to check retry implementation
  • Interrupt requests to td-api to count request calls or check the content in request
td operator -> littleshot proxy -> Treasure Data API
                  ^ count the number of POST /v3/job/issue and so on

But Treasure Data API doesn’t accept any HTTP request now. So I’ve tried using HTTPS requests. But the verification of request call count in the test code failed.
That’s because littleshot proxy ’s request interruption doesn’t work for HTTPS request. It creates a SSL tunnel after handling CONNECT method and doesn’t touch following request calls. With current littleshot proxy, we can’t achieve the original verification in the test code, so I’ll comment out them since current situation blocks our development. But we need to take care of it soon because it’s kinda lack of testing.

@komamitsu komamitsu changed the base branch from master to v0_11 April 7, 2021 09:10
@komamitsu komamitsu closed this Apr 7, 2021
@komamitsu komamitsu reopened this Apr 7, 2021
@komamitsu komamitsu changed the title [WIP] Fix td endpoint protocol Fix td endpoint protocol Apr 9, 2021
@komamitsu komamitsu changed the title Fix td endpoint protocol Fix td operators tests by using HTTPS Apr 9, 2021
@komamitsu komamitsu merged commit 2a6516e into v0_11 Apr 9, 2021
@komamitsu komamitsu deleted the fix-td-endpoint-protocol branch April 9, 2021 07:16
yoyama pushed a commit to yoyama/digdag that referenced this pull request May 21, 2021
* Fix a faling test

* For investigation

* Revert

* Fix

* Use HTTPS for td operators tests

* Stop using HTTP proxy

* Stop relaying on littleshot req interruptions
@szyn szyn added this to the v0.11.0 milestone Jun 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants