feat:support of pass throught hosts #1397

shivamsouravjha · 2024-01-18T12:18:24Z

Related Issue

Info about Issue or bug

Closes: #1382

Describe the changes you've made

filter url (same method same url, same url no method, different method same url)
Ignore host for test filter, with host - path: ".*/get$"
stubs for reqres path with w/o regex checked

Genrated Yaml

record:
    path: ""
    # mandatory
    command: ""
    proxyport: 0
    containerName: ""
    networkName: ""
    delay: 5
    buildDelay: 30s
    tests:
        filters:
            - path: ""
              urlMethods: []
              headers: {}
              host: ""
    stubs:
        filters:
            - path: ""
              host: "reqres.in"
              port: 0
test:
    path: ""
    # mandatory
    command: ""
    proxyport: 0
    containerName: ""
    networkName: ""
    # example: "test-set-1": ["test-1", "test-2", "test-3"]
    selectedTests:
    # to use globalNoise, please follow the guide at the end of this file.
    globalNoise:
        global:
            body: {}
            header: {}
    delay: 5
    buildDelay: 30s
    apiTimeout: 5
    stubs:
        filters:
            - path: ""
              host: "reqres.in"
              port: 0
    withCoverage: false
    coverageReportPath: ""
    # Example on using tests
    # tests: 
    #   filters:
    #    - path: "/user/app"
    #      urlMethods: ["GET"]
    #      headers: {
    #        "^asdf*": "^test"
    #      }
    #      host: "dc.services.visualstudio.com"
    # Example on using stubs
    # stubs: 
    #   filters:
    #    - path: "/user/app"
    #      port: 8080
    #    - port: 8081
    #    - host: "dc.services.visualstudio.com"
    #    - port: 8081
    #      host: "dc.services.visualstudio.com"
    #      path: "/user/app"
    #
    # Example on using globalNoise
    # globalNoise: 
    #    global:
    #      body: {
    #         # to ignore some values for a field, 
    #         # pass regex patterns to the corresponding array value
    #         "url": ["https?://\S+", "http://\S+"],
    #      }
    #      header: {
    #         # to ignore the entire field, pass an empty array
    #         "Date": [],
    #       }
    #     # to ignore fields or the corresponding values for a specific test-set,
    #     # pass the test-set-name as a key to the "test-sets" object and
    #     # populate the corresponding "body" and "header" objects 
    #     test-sets:
    #       test-set-1:
    #         body: {
    #           # ignore all the values for the "url" field
    #           "url": []
    #         }
    #         header: { 
    #           # we can also pass the exact value to ignore for a field
    #           "User-Agent": ["PostmanRuntime/7.34.0"]
    #         }

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Code style update (formatting, local variables)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist:

My code follows the style guidelines of this project.
I have performed a self-review of my own code.
I have commented my code, particularly in hard-to-understand areas.
I have made corresponding changes to the documentation.
My changes generate no new warnings.
I have added tests that prove my fix is effective or that my feature works.
New and existing unit tests pass locally with my changes.

pkg/models/const.go

charankamarapu

Please address the comments

charankamarapu · 2024-01-21T14:57:58Z

pkg/service/generateConfig/generateConfig.go

  withCoverage: false
  coverageReportPath: ""
-  `
+  # Example on using tests


I think this should be added in config guide. You can see how is the config guide attached to final output.

pkg/hooks/loader.go

charankamarapu · 2024-01-21T15:02:58Z

cmd/record.go

 			if err != nil {
 				if err == errFileNotFound {
-					r.logger.Info("continuing without configuration file because file not found")
+					r.logger.Info("Keploy config not found, using default config")


This log is wrong if keploy config is present we will not use default config. We consider only flags and continue with flags.

Please provide an update on this

cmd/record.go

cmd/test.go

pkg/platform/yaml/yaml.go

pkg/proxy/integrations/httpparser/httpparser.go

cmd/record.go

charankamarapu

Please address the comments

charankamarapu · 2024-01-26T19:09:21Z

pkg/hooks/loader.go

@@ -141,6 +142,16 @@ func (h *Hook) GetProxyPort() uint32 {
 	return h.proxyPort
 }

+func (h *Hook) GetProxyHost() models.Stubs {


Rename the function. Current name provide wrong info

charankamarapu · 2024-01-26T19:09:39Z

pkg/hooks/loader.go

+	return h.passThroughHosts
+}
+
+func (h *Hook) SetProxyHosts(passThroughHosts []models.Filters) {


Rename the function

charankamarapu · 2024-01-26T19:14:10Z

pkg/platform/yaml/yaml.go

+		urlMatched = true
+	}
+
+	if len(urlMethods) != 0 {


Looks like this logic has flaw lets say user has given path - x and port -y . even if there is a request with path - z and port - y it will be matched and considered as passthrough.

seems like this selected code block deals with some other logic.

charankamarapu · 2024-01-26T19:16:52Z

pkg/proxy/integrations/httpparser/httpparser.go

@@ -31,17 +32,17 @@ type HttpParser struct {
 }

 // ProcessOutgoing implements proxy.DepInterface.
-func (http *HttpParser) ProcessOutgoing(request []byte, clientConn, destConn net.Conn, ctx context.Context) {
+func (http *HttpParser) ProcessOutgoing(request []byte, clientConn, destConn net.Conn, ctx context.Context, sourcePort int) {


Can't you extract port from clientConn or destConn..?

will increase computation time, better to use a precalculated value

utils/utils.go

charankamarapu

please address the comments, rest looks fine.

charankamarapu · 2024-01-29T19:17:40Z

cmd/record.go

 			if err != nil {
 				if err == errFileNotFound {
-					r.logger.Info("continuing without configuration file because file not found")
+					r.logger.Info("Keploy config not found, ontinuing without configuration")


please correct the spelling error.

charankamarapu · 2024-01-29T19:20:25Z

cmd/test.go

 			if err != nil {
 				if err == errFileNotFound {
-					t.logger.Info("continuing without configuration file because file not found")
+					t.logger.Info("Keploy config not found, using default config")


Please provide an update on this.

charankamarapu · 2024-01-29T19:35:03Z

pkg/platform/yaml/yaml.go

+				return err, false
+			}
+			headerValueMatch := regex.MatchString(headerNameValue)
+			if headerNameMatch || headerValueMatch {


Shouldn't this be a and condition..?

earlier I thought of using either header name or value to be independent, will update it so that the combination is unique(i.e, the and condition)

pkg/proxy/integrations/httpparser/httpparser.go

charankamarapu · 2024-01-29T19:46:33Z

pkg/proxy/integrations/httpparser/httpparser.go

+
+			}
+			for _, filter := range portPassThrough {
+				regex, err := regexp.Compile(filter.Path)


Is it not possible in config to provide path without port. Why is path and port are combined?

We separate only port and combination of port +host in service layer. then in this layer we separate port+ host and only host. thus there are two loops too.

charankamarapu · 2024-01-29T19:47:27Z

pkg/proxy/integrations/httpparser/httpparser.go

+				}
+				matches := regex.MatchString(req.URL.String())
+				if matches && host != "" || req.Host == host {
+					passthroughHost = true


Shouldn't we have a break statement here..?

charankamarapu · 2024-01-29T19:48:14Z

pkg/proxy/integrations/httpparser/httpparser.go

-		if req.Host == host {
+	passThrough := models.PassThroughHosts
+	portPassThrough := []models.Filters{}
+	for _, filters := range h.GetPassThroughHosts().Filters {


similarly here too.

charankamarapu

Ship it!

shivamsouravjha force-pushed the shivam/passthrough branch from 2a1c7a4 to 8f35c38 Compare January 18, 2024 12:25

shivamsouravjha requested review from charankamarapu and Sarthak160 January 19, 2024 10:36

github-advanced-security bot found potential problems Jan 19, 2024

View reviewed changes

pkg/models/const.go Fixed Show fixed Hide fixed

shivamsouravjha force-pushed the shivam/passthrough branch from e78ff66 to eeab5a0 Compare January 19, 2024 12:22

charankamarapu reviewed Jan 22, 2024

View reviewed changes

shivamsouravjha force-pushed the shivam/passthrough branch from eeab5a0 to a6afe99 Compare January 22, 2024 10:05

charankamarapu self-requested a review January 23, 2024 11:10

charankamarapu reviewed Jan 26, 2024

View reviewed changes

shivamsouravjha force-pushed the shivam/passthrough branch 2 times, most recently from 70ef6a4 to 77cb6cd Compare January 29, 2024 10:37

charankamarapu reviewed Jan 29, 2024

View reviewed changes

shivamsouravjha added 17 commits January 30, 2024 02:50

feat:support of pass throught hosts

3dbf599

refactor:filter, stub and passthrough

7582257

refactor:passthroughthosts

9d21c52

refactor: pipeline code

4f34372

refactor:PassThroughHosts

0f24900

feat:support of pass throught hosts

8c1d138

refactor:filter, stub and passthrough

477870d

refactor:parse and loader

06b1aee

fix:yam

4f175c4

udpate:append

cf5524e

refactor:log

4e292eb

refactor:url match

c217d71

refactor:logic to update passthrough hosts

ed07422

refactor:update function name,move source port logic

d084885

refactor:removing filters from tests

0a44c8d

refactor:logs and config

9c95727

fix:merge errors

0c7bff4

shivamsouravjha force-pushed the shivam/passthrough branch from ab553d6 to 0c7bff4 Compare January 30, 2024 02:57

shivamsouravjha added 3 commits January 30, 2024 03:08

fix:selected tests in config

478abe5

refactor:passthrough host and path

3bbadcb

update:name of variable

56fbb52

charankamarapu approved these changes Jan 30, 2024

View reviewed changes

charankamarapu merged commit 7e44c3b into main Jan 30, 2024
13 checks passed

charankamarapu deleted the shivam/passthrough branch January 30, 2024 06:01

github-actions bot locked and limited conversation to collaborators Jan 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat:support of pass throught hosts #1397

feat:support of pass throught hosts #1397

shivamsouravjha commented Jan 18, 2024 •

edited

charankamarapu left a comment

charankamarapu Jan 21, 2024

charankamarapu Jan 21, 2024

charankamarapu Jan 26, 2024

charankamarapu left a comment

charankamarapu Jan 26, 2024

charankamarapu Jan 26, 2024

charankamarapu Jan 26, 2024

shivamsouravjha Jan 29, 2024

charankamarapu Jan 26, 2024

shivamsouravjha Jan 28, 2024

charankamarapu left a comment

charankamarapu Jan 29, 2024

charankamarapu Jan 29, 2024

charankamarapu Jan 29, 2024

shivamsouravjha Jan 30, 2024

charankamarapu Jan 29, 2024

shivamsouravjha Jan 30, 2024

charankamarapu Jan 29, 2024

shivamsouravjha Jan 30, 2024

charankamarapu Jan 29, 2024

charankamarapu left a comment

feat:support of pass throught hosts #1397

feat:support of pass throught hosts #1397

Conversation

shivamsouravjha commented Jan 18, 2024 • edited

Related Issue

Describe the changes you've made

Type of change

Checklist:

charankamarapu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charankamarapu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charankamarapu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charankamarapu left a comment

Choose a reason for hiding this comment

shivamsouravjha commented Jan 18, 2024 •

edited