Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ability to turn off proxy based on request url #2065

Closed
harshmaur opened this issue Sep 5, 2023 · 1 comment · Fixed by #2392
Closed

Ability to turn off proxy based on request url #2065

harshmaur opened this issue Sep 5, 2023 · 1 comment · Fixed by #2392
Assignees
@barjin
Labels
feature Issues that represent new features or improvements to existing features. t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@harshmaur
Copy link

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/core

Feature

We can currently configure a list of proxies and we can use a method to return a particular proxy url. However, it would be great to have the ability to turn off proxy completely dynamically.

Motivation

This is particularly useful where proxy can be used to load the website but we dont need proxy to load the scripts, css and images via proxy and usually these page resources take a lot of time and consume a lot of proxy bandwidth.

Ideal solution or implementation, and any additional constraints

In the newUrlFunction if we return undefined, then the proxy is not used at all.

Alternative solutions or implementations

No response

Other context

No response
@harshmaur harshmaur added the feature Issues that represent new features or improvements to existing features. label Sep 5, 2023
@fnesveda fnesveda added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 6, 2023
@harshmaur
Copy link
Author

I am happy to create a PR to help support this, I just need to know where to look and what to investigate there.

@barjin barjin mentioned this issue Sep 20, 2023
Closed
@barjin barjin mentioned this issue Feb 21, 2024
Merged
@barjin barjin mentioned this issue Mar 26, 2024
Merged
barjin added a commit that referenced this issue Apr 4, 2024
@barjin
feat: better newUrlFunction for ProxyConfiguration (#2392)
330598b 
Based on changes from #2348 , this PR simplifies the proxy handling in
the browser crawlers and makes those more intuitive.

Closes #2065
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@barjin barjin

Labels
feature Issues that represent new features or improvements to existing features. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Milestone
No milestone
3 participants
@fnesveda @harshmaur @barjin