Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Origin/proxy v2 #50

Draft
wants to merge 194 commits into
base: main
Choose a base branch
from
Draft

Origin/proxy v2 #50

wants to merge 194 commits into from

Conversation

deoxykev
Copy link
Contributor

@deoxykev deoxykev commented Nov 28, 2023

WIP

Improvements

  • Complete rewrite of proxy logic
  • Support all http methods
  • Flexible and modular API
  • Tons of prebuilt request/response modules
  • Support for spoofing JA3 hashes
  • Robust URL rewriting
    • HTTP URL rewriter now uses http tokenizer and handles many edge cases
    • Implemented client-side JS URL rewriter by patching JS runtime
  • JS execution time can be specified (beforeDOMLoad, afterDOMLoad, afterDOMIdle)
  • Lots of other stuff

Usage

import (
   "ladder/proxychain"
   rx "ladder/proxychain/requestmodifers"
   tx "ladder/proxychain/responsemodifers"

   "github.com/gofiber/fiber/v2"
)

proxychain.
 NewProxyChain().
 SetFiberCtx(c).
 SetDebugLogging(opts.Verbose).
 SetRequestModifications(
  	rx.MasqueradeAsFacebookBot(),
  	rx.DeleteOutgoingCookies(),
 ).
 AddResponseModifications(
  	tx.BypassCORS(),
  	tx.BypassContentSecurityPolicy(),
  	tx.DeleteIncomingCookies(),
  	tx.RewriteHTMLResourceURLs(),
  	tx.PatchDynamicResourceURLs(),
 ).
 Execute()

Todo:

  • CSS rewriter
  • Ruleset support
  • Unit tests
  • Bug fixes

New Request Modifiers

"ladder/proxychain/requestmodifers"

  • DeleteOutgoingCookie
  • DeleteOutgoingCookies
  • DeleteOutgoingCookiesExcept
  • DeleteRequestHeader
  • ForwardRequestHeaders
  • HideOrigin
  • HideReferrer
  • MasqueradeAsBaiduBot
  • MasqueradeAsBingBot
  • MasqueradeAsDuckDuckBot
  • MasqueradeAsFacebookBot
  • MasqueradeAsGoogleBot
  • MasqueradeAsWaybackMachineBot
  • MasqueradeAsYahooBot
  • MasqueradeAsYandexBot
  • ModifyDomainWithRegex
  • ModifyPathWithRegex
  • ModifyQueryParams
  • RequestArchiveIs
  • RequestGoogleCache
  • RequestWaybackMachine
  • ResolveWithGoogleDoH
  • SetOutgoingCookie
  • SetOutgoingCookies
  • SetRequestHeader
  • SpoofJA3fingerprint
  • SpoofJA3fingerprintWithProxy
  • SpoofOrigin
  • SpoofReferrer
  • SpoofReferrerFromBaiduSearch
  • SpoofReferrerFromBingSearch
  • SpoofReferrerFromGoogleSearch
  • SpoofReferrerFromLinkedInPost
  • SpoofReferrerFromNaverSearch
  • SpoofReferrerFromPinterestPost
  • SpoofReferrerFromQQPost
  • SpoofReferrerFromRedditPost
  • SpoofReferrerFromTumblrPost
  • SpoofReferrerFromTwitterPost
  • SpoofReferrerFromVkontaktePost
  • SpoofReferrerFromWeiboPost
  • SpoofUserAgent
  • SpoofXForwardedFor

New Response Modifiers

"ladder/proxychain/responsemodifers"

  • BypassCORS
  • BypassContentSecurityPolicy
  • DeleteIncomingCookies
  • DeleteIncomingCookiesExcept
  • DeleteResponseHeader
  • ForwardResponseHeaders
  • InjectScriptAfterDOMContentLoaded
  • InjectScriptAfterDOMIdle
  • InjectScriptBeforeDOMContentLoaded
  • PatchDynamicResourceURLs
  • RewriteHTMLResourceURLs
  • SetContentSecurityPolicy
  • SetIncomingCookie
  • SetIncomingCookies
  • SetResponseHeader

…and proxified URLs; direct injection from golang
…ers; improve html rewriter edge case handling
# Aider chat conversation:

USER: Modify proxychain.go such that there is a public API to add a RequestModification to fire only once.

ASSISTANT: To make this change we need to modify `proxychain/proxychain.go` to:

1. Add a new field `onceRequestModifications` in the `ProxyChain` struct to store the request modifications that should only fire once.
2. Add a new method `AddOnceRequestModification` to add a request modification that should only fire once.
3. Update the `_execute` method to apply the `onceRequestModifications` and clear them after they have been applied.

Here are the *SEARCH/REPLACE* blocks:

proxychain/proxychain.go
<source>
<<<<<<< SEARCH
type ProxyChain struct {
	Context              *fiber.Ctx
	Client               *http.Client
	Request              *http.Request
	Response             *http.Response
	requestModifications []RequestModification
	resultModifications  []ResponseModification
	htmlTokenRewriters   []rr.IHTMLTokenRewriter
	Ruleset              *ruleset.RuleSet
	debugMode            bool
	abortErr             error
}
=======
type ProxyChain struct {
	Context                 *fiber.Ctx
	Client                  *http.Client
	Request                 *http.Request
	Response                *http.Response
	requestModifications    []RequestModification
	onceRequestModifications []RequestModification
	resultModifications     []ResponseModification
	htmlTokenRewriters      []rr.IHTMLTokenRewriter
	Ruleset                 *ruleset.RuleSet
	debugMode               bool
	abortErr                error
}
>>>>>>> REPLACE
</source>

proxychain/proxychain.go
<source>
<<<<<<< SEARCH
// AddRequestModifications sets the ProxyChain's request modifers
// the modifier will not fire until ProxyChain.Execute() is run.
func (chain *ProxyChain) AddRequestModifications(mods ...RequestModification) *ProxyChain {
	chain.requestModifications = append(chain.requestModifications, mods...)
	return chain
}
=======
// AddRequestModifications adds more request modifers to the ProxyChain
// the modifier will not fire until ProxyChain.Execute() is run.
func (chain *ProxyChain) AddRequestModifications(mods ...RequestModification) *ProxyChain {
	chain.requestModifications = append(chain.requestModifications, mods...)
	return chain
}

// AddOnceRequestModification adds a request modifier to the ProxyChain that should only fire once
// the modifier will not fire until ProxyChain.Execute() is run and will be removed after it has been applied.
func (chain *ProxyChain) AddOnceRequestModification(mod ...RequestModification) *ProxyChain {
	chain.onceRequestModifications = append(chain.onceRequestModifications, mod...)
	return chain
}
>>>>>>> REPLACE
</source>

proxychain/proxychain.go
<source>
<<<<<<< SEARCH
// _execute sends the request for the ProxyChain and returns the raw body only
// the caller is responsible for returning a response back to the requestor
// the caller is also responsible for calling chain._reset() when they are done with the body
func (chain *ProxyChain) _execute() (io.Reader, error) {
	if chain.validateCtxIsSet() != nil || chain.abortErr != nil {
		return nil, chain.abortErr
	}
	if chain.Request == nil {
		return nil, errors.New("proxychain request not yet initialized")
	}
	if chain.Request.URL.Scheme == "" {
		return nil, errors.New("request url not set or invalid. Check ProxyChain ReqMods for issues")
	}

	// Apply requestModifications to proxychain
	for _, applyRequestModificationsTo := range chain.requestModifications {
		err := applyRequestModificationsTo(chain)
		if err != nil {
			return nil, chain.abort(err)
		}
	}
=======
// _execute sends the request for the ProxyChain and returns the raw body only
// the caller is responsible for returning a response back to the requestor
// the caller is also responsible for calling chain._reset() when they are done with the body
func (chain *ProxyChain) _execute() (io.Reader, error) {
	if chain.validateCtxIsSet() != nil || chain.abortErr != nil {
		return nil, chain.abortErr
	}
	if chain.Request == nil {
		return nil, errors.New("proxychain request not yet initialized")
	}
	if chain.Request.URL.Scheme == "" {
		return nil, errors.New("request url not set or invalid. Check ProxyChain ReqMods for issues")
	}

	// Apply requestModifications to proxychain
	for _, applyRequestModificationsTo := range chain.requestModifications {
		err := applyRequestModificationsTo(chain)
		if err != nil {
			return nil, chain.abort(err)
		}
	}

	// Apply onceRequestModifications to proxychain and clear them
	for _, applyOnceRequestModificationsTo := range chain.onceRequestModifications {
		err := applyOnceRequestModificationsTo(chain)
		if err != nil {
			return nil, chain.abort(err)
		}
	}
	chain.onceRequestModifications = nil
>>>>>>> REPLACE
</source>
# Aider chat conversation:

USER: Modify proxychain.go such that there is a public API to add a ResponseModification to fire only once.

ASSISTANT: To make this change we need to modify `proxychain/proxychain.go` to:

1. Add a new field `onceResultModifications` in the `ProxyChain` struct to store the response modifications that should only fire once.
2. Add a new method `AddOnceResponseModification` to add a response modification that should only fire once.
3. Update the `_execute` method to apply the `onceResultModifications` and clear them after they have been applied.

Here are the *SEARCH/REPLACE* blocks:

proxychain/proxychain.go
<source>
<<<<<<< SEARCH
onceRequestModifications []RequestModification
resultModifications     []ResponseModification
=======
onceRequestModifications []RequestModification
onceResultModifications []ResponseModification
resultModifications     []ResponseModification
>>>>>>> REPLACE
</source>

proxychain/proxychain.go
<source>
<<<<<<< SEARCH
// AddOnceRequestModification adds a request modifier to the ProxyChain that should only fire once
// the modifier will not fire until ProxyChain.Execute() is run and will be removed after it has been applied.
func (chain *ProxyChain) AddOnceRequestModification(mod ...RequestModification) *ProxyChain {
	chain.onceRequestModifications = append(chain.onceRequestModifications, mod...)
	return chain
}
=======
// AddOnceRequestModification adds a request modifier to the ProxyChain that should only fire once
// the modifier will not fire until ProxyChain.Execute() is run and will be removed after it has been applied.
func (chain *ProxyChain) AddOnceRequestModification(mod ...RequestModification) *ProxyChain {
	chain.onceRequestModifications = append(chain.onceRequestModifications, mod...)
	return chain
}

// AddOnceResponseModification adds a response modifier to the ProxyChain that should only fire once
// the modifier will not fire until ProxyChain.Execute() is run and will be removed after it has been applied.
func (chain *ProxyChain) AddOnceResponseModification(mod ...ResponseModification) *ProxyChain {
	chain.onceResultModifications = append(chain.onceResultModifications, mod...)
	return chain
}
>>>>>>> REPLACE
</source>

proxychain/proxychain.go
<source>
<<<<<<< SEARCH
// Apply onceRequestModifications to proxychain and clear them
for _, applyOnceRequestModificationsTo := range chain.onceRequestModifications {
	err := applyOnceRequestModificationsTo(chain)
	if err != nil {
		return nil, chain.abort(err)
	}
}
chain.onceRequestModifications = nil
=======
// Apply onceRequestModifications and onceResultModifications to proxychain and clear them
for _, applyOnceRequestModificationsTo := range chain.onceRequestModifications {
	err := applyOnceRequestModificationsTo(chain)
	if err != nil {
		return nil, chain.abort(err)
	}
}
chain.onceRequestModifications = nil

for _, applyOnceResultModificationsTo := range chain.onceResultModifications {
	err := applyOnceResultModificationsTo(chain)
	if err != nil {
		return nil, chain.abort(err)
	}
}
chain.onceResultModifications = nil
>>>>>>> REPLACE
</source>
@deoxykev deoxykev marked this pull request as draft November 28, 2023 00:11
deoxykev and others added 30 commits December 9, 2023 08:37
Render error page when accept header is text/html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants