proxytest: proxy HTTPS request using MITM #5802

AndersonQ · 2024-10-16T09:18:02Z

What does this PR do?

It extends proxytest to support proxying HTTPS requests.

The proxy can now act as a MITM (man-in-the-middle) proxy for HTTPS, generating certificates on the fly for the target host.
The proxy assumes it should proxy an HTTPS request if it receives a CONNECT requests.

Why is it important?

It'll allow us to create integration tests for mTLS scenarios

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~[ ] I have added an entry in ./changelog/fragments using the changelog tool~~
~~[ ] I have added an integration test or an E2E test~~

Disruptive User Impact

None

How to test this PR locally

Run TestRunHTTPSProxy passing the example build tag and follow the instructions to perform a cRUL request using the proxytest:

run the test: go test -tags example -v -run TestRunHTTPSProxy$ ./testing/proxytest
run the provided curl command

Example:

$ go test -tags example -v -run TestRunHTTPSProxy$ ./testing/proxytest
=== RUN   TestRunHTTPSProxy
    proxytest_test.go:451: target server running on https://127.0.0.1:34355
    proxytest.go:100: [proxy-https] time=2024-10-16T11:14:33.982+02:00 level=INFO source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:236 msg="running on https://127.0.0.1:40241 -> https://localhost:40241"
    proxytest_test.go:473: proxy running on https://localhost:40241
    proxytest_test.go:479: make request to "https://not-a-server.co" using proxy "https://localhost:40241"
    proxytest_test.go:481: curl \
        --proxy-cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy-cert /tmp/TestRunHTTPSProxy3155179289/001/clientCert.crt \
        --proxy-key /tmp/TestRunHTTPSProxy3155179289/001/clientCert.key \
        --cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy https://127.0.0.1:40241 \
        https://not-a-server.co
    proxytest_test.go:496: CTRL+C to stop
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=INFO source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:183 msg="STARTING - CONNECT '//not-a-server.co:443' HTTP/1.1 127.0.0.1:40496"
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:22 msg="handling CONNECT" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.046+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:34 msg="hijacked request" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.092+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/proxytest.go:282 msg="original URL: https://not-a-server.co:443/, new URL: https://127.0.0.1:34355/"
    proxytest.go:100: [proxy-https] time=2024-10-16T11:15:19.097+02:00 level=DEBUG source=/home/ainsoph/devel/github.com/elastic/elastic-agent/testing/proxytest/https.go:129 msg="EOF reached, finishing HTTPS handler" req_id=8a3b5c0f-50c0-40d1-84af-59412ed8bdc3
    proxytest.go:100: [proxy-https] [8a3b5c0f-50c0-40d1-84af-59412ed8bdc3] DONE 0 - CONNECT //not-a-server.co:443 HTTP/1.1 127.0.0.1:40496
^Csignal: interrupt

$ curl \
        --proxy-cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy-cert /tmp/TestRunHTTPSProxy3155179289/001/clientCert.crt \
        --proxy-key /tmp/TestRunHTTPSProxy3155179289/001/clientCert.key \
        --cacert /tmp/TestRunHTTPSProxy3155179289/001/proxyCA.crt \
        --proxy https://127.0.0.1:40241 \
        https://not-a-server.co
It works!

Related issues

Closes Improve test proxy/mock fleet server to proxy HTTPS connections #4903

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

mergify · 2024-10-16T09:18:43Z

This pull request does not have a backport label. Could you fix it @AndersonQ? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-10-16T09:18:43Z

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

AndersonQ · 2024-10-16T09:21:46Z

go.mod

 	github.com/elastic/elastic-agent-autodiscover v0.9.0
 	github.com/elastic/elastic-agent-client/v7 v7.16.0
-	github.com/elastic/elastic-agent-libs v0.12.1
+	github.com/elastic/elastic-agent-libs v0.12.2-0.20241016085313-16569381887f


once elastic/elastic-agent-libs#240 is merged it will be updated to v0.12.2

elasticmachine · 2024-10-16T10:50:49Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

swiatekm

Looks correct overall, had some questions about writing HTTP requests by hand.

testing/proxytest/https.go

swiatekm · 2024-10-21T12:43:44Z

testing/proxytest/https.go

+	resp := bytes.Buffer{}
+	resp.WriteString(fmt.Sprintf("HTTP/1.1 %d %s\r\n",
+		statusCode, http.StatusText(statusCode)))
+	resp.WriteString("Content-Type: text/plain\r\n")
+	resp.WriteString(fmt.Sprintf("Content-Length: %d\r\n", len(body)))
+	resp.WriteString("\r\n")
+	if len(body) > 0 {
+		resp.Write(body)
+	}


This would look better if we instead created a http.Response and written it to the buffer.

AndersonQ · 2024-10-25T09:50:53Z

@swiatekm I've ~~finally~~ made the changes you requested. Sorry I ended up force pushing them

The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server.

…https-proxytest

elastic-sonarqube · 2024-10-28T14:53:06Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server. (cherry picked from commit ad041d4)

The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server. (cherry picked from commit ad041d4) Co-authored-by: Anderson Queiroz <anderson.queiroz@elastic.co>

The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server. (cherry picked from commit ad041d4) Co-authored-by: Anderson Queiroz <anderson.queiroz@elastic.co> (cherry picked from commit ec75170)

The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server. (cherry picked from commit ad041d4)

The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server. (cherry picked from commit ad041d4) Co-authored-by: Anderson Queiroz <anderson.queiroz@elastic.co>

AndersonQ requested a review from a team as a code owner October 16, 2024 09:18

AndersonQ requested review from leehinman, pchila and swiatekm October 16, 2024 09:18

AndersonQ self-assigned this Oct 16, 2024

AndersonQ added skip-changelog Testing labels Oct 16, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 16, 2024

AndersonQ force-pushed the 4903-https-proxytest branch from 34296a6 to 5cb739b Compare October 16, 2024 09:19

AndersonQ commented Oct 16, 2024

View reviewed changes

AndersonQ force-pushed the 4903-https-proxytest branch 4 times, most recently from b80b705 to 89f4a5d Compare October 16, 2024 09:54

pierrehilbert added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Oct 16, 2024

AndersonQ force-pushed the 4903-https-proxytest branch 5 times, most recently from fd3a299 to eddcaab Compare October 17, 2024 09:41

swiatekm reviewed Oct 21, 2024

View reviewed changes

AndersonQ force-pushed the 4903-https-proxytest branch 4 times, most recently from b295b2a to b2614b4 Compare October 25, 2024 09:50

proxytest: proxy HTTPS request using MITM

a53b932

The proxytest now can proxy HTTPS requests using a men in the middle (MITM) approach to allow to fully control the requests between the proxy and the target server.

AndersonQ force-pushed the 4903-https-proxytest branch from b2614b4 to a53b932 Compare October 25, 2024 10:27

AndersonQ added 2 commits October 25, 2024 17:01

Merge branch 'main' into 4903-https-proxytest

0787803

Merge branch 'main' into 4903-https-proxytest

b4faaf0

swiatekm approved these changes Oct 28, 2024

View reviewed changes

AndersonQ added 2 commits October 28, 2024 10:42

fix test

546654f

Merge remote-tracking branch 'origin/4903-https-proxytest' into 4903-…

85cd04e

…https-proxytest

AndersonQ enabled auto-merge (squash) October 28, 2024 09:43

Merge branch 'main' into 4903-https-proxytest

7b73730

AndersonQ merged commit ad041d4 into elastic:main Oct 28, 2024
14 checks passed

mergify bot mentioned this pull request Oct 28, 2024

[8.x](backport #5802) proxytest: proxy HTTPS request using MITM #5867

Merged

3 tasks

mauri870 mentioned this pull request Oct 28, 2024

[Flaky Test]: TestHTTPSProxy – data race #5869

Closed

AndersonQ deleted the 4903-https-proxytest branch October 29, 2024 06:37

mergify bot mentioned this pull request Nov 4, 2024

[8.16](backport #5867) [8.x](backport #5802) proxytest: proxy HTTPS request using MITM #5925

Closed

3 tasks

AndersonQ added the backport-8.16 Automated backport with mergify label Nov 4, 2024

mergify bot mentioned this pull request Nov 4, 2024

[8.16](backport #5802) proxytest: proxy HTTPS request using MITM #5928

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

proxytest: proxy HTTPS request using MITM #5802

proxytest: proxy HTTPS request using MITM #5802

Uh oh!

AndersonQ commented Oct 16, 2024 •

edited

Loading

Uh oh!

mergify bot commented Oct 16, 2024

Uh oh!

mergify bot commented Oct 16, 2024

Uh oh!

AndersonQ Oct 16, 2024

Uh oh!

elasticmachine commented Oct 16, 2024

Uh oh!

swiatekm left a comment

Uh oh!

Uh oh!

swiatekm Oct 21, 2024

Uh oh!

AndersonQ commented Oct 25, 2024

Uh oh!

elastic-sonarqube bot commented Oct 28, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

proxytest: proxy HTTPS request using MITM #5802

proxytest: proxy HTTPS request using MITM #5802

Uh oh!

Conversation

AndersonQ commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

Uh oh!

mergify bot commented Oct 16, 2024

Uh oh!

mergify bot commented Oct 16, 2024

Uh oh!

AndersonQ Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 16, 2024

Uh oh!

swiatekm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

swiatekm Oct 21, 2024

Choose a reason for hiding this comment

Uh oh!

AndersonQ commented Oct 25, 2024

Uh oh!

elastic-sonarqube bot commented Oct 28, 2024

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AndersonQ commented Oct 16, 2024 •

edited

Loading