Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Depop 403 #47

Closed
Geo0010 opened this issue Feb 14, 2024 · 11 comments · Fixed by #52
Closed

Depop 403 #47

Geo0010 opened this issue Feb 14, 2024 · 11 comments · Fixed by #52
Assignees
Labels
bug Something isn't working

Comments

@Geo0010
Copy link

Geo0010 commented Feb 14, 2024

When running the script for Depop users, the request to the Depop api results in an authorization error (403 forbidden). I’ve tried different headers, vpns, and cloudscraper with no luck. Vinted still works without issue. Any advice would be appreciated. Thanks!

@Relax594
Copy link

Relax594 commented Feb 14, 2024

They've added CloudFlare and you can't access the website without JavaScript enabled anymore. There might be additional protection layer somewhere which I haven't noticed yet.

There is also this in the original request headers now:
Access-Control-Request-Headers: depop-device-id,depop-search-id,depop-session-id,x-cached-sizes

grafik

@Gertje823 Gertje823 self-assigned this Feb 14, 2024
@Gertje823 Gertje823 added the bug Something isn't working label Feb 14, 2024
Gertje823 added a commit that referenced this issue Feb 14, 2024
Depop API now requires referer header.
@Gertje823
Copy link
Owner

Thanks for opening this issue. Apparently the Depop api requires a referer request header now.
I updated the script and it should work now.

@mkyrganov
Copy link

mkyrganov commented Feb 16, 2024

doesnt work

@Relax594
Copy link

dont work

for me it did yesterday and the day before. Haven't tested today.

@mkyrganov
Copy link

dont work

for me it did yesterday and the day before. Haven't tested today.

Снимок экрана 2024-02-16 в 16 48 51

Already at this point Im getting 403

@Gertje823 Gertje823 reopened this Feb 16, 2024
@Gertje823
Copy link
Owner

I can indeed reproduce the error you are getting. Looks like sometimes it works and sometimes it is detected by Cloudflare. I am testing different Cloudscraper configs to see if I can bypass the Cloudflare detection that way.

@mkyrganov
Copy link

mkyrganov commented Feb 16, 2024

I can indeed reproduce the error you are getting. Looks like sometimes it works and sometimes it is detected by Cloudflare. I am testing different Cloudscraper configs to see if I can bypass the Cloudflare detection that way.

`
import asyncio
import cloudscraper

async def main():

headers = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Accept-Encoding": "gzip,deflate,br",
    "Accept-Language": "en-US,en;q=0.9",
    "Cache-Control": "max-age=0",
    "Dnt": "1",
    "Sec-Ch-Ua": '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"Windows"',
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "same-origin",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

headers2 = {
    "Accept": "application/json, text/plain, */*",
    "Accept-Encoding": "gzip,deflate,br",
    "Accept-Language": "en-US,en;q=0.9",
    "Origin": "https://www.depop.com",
    "Referer": "https://www.depop.com/",
    "Sec-Ch-Ua": '"Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"Windows"',
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "same-site",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

s = cloudscraper.create_scraper()

r = s.get(
    "https://www.depop.com/search/?q=gucci", 
    proxies = {"http": "", "https": ""}, headers=headers
)

print("SITE PAGE", r.status_code) 

r = s.get(
    "https://webapi.depop.com/api/v2/search/products/?what=gucci&itemsPerPage=24&country=us&currency=USD&sort=relevance", proxies = {"http": "", "https": ""}, headers=headers2
)
print("API PAGE", r.status_code)  # this work  

if name == "main":
asyncio.run(main())
`

these headers (headers2) work for the webapi domain, but the main domain doesnt work

Gertje823 added a commit that referenced this issue Feb 16, 2024
Added cloudscraper browser parameters to potentially solve #47
@Gertje823
Copy link
Owner

Gertje823 commented Feb 16, 2024

I added browser parameters for couldscraper and I was able to scrape some profiles without any problems. Could you please checkout https://github.com/Gertje823/Vinted-Scraper/tree/47-depop-403 and let me know if this is also working for you guys?

@Relax594
Copy link

Relax594 commented Apr 27, 2024

Hey @Gertje823 sorry for coming back late. It worked fine with using the "chrome" browser parameters until like a few days ago. Now that also ends up in a lot of 403 requests. Using your browser parameters didn't change anything.

@Gertje823
Copy link
Owner

@Relax594 @mkyrganov @Geo0010 I haven't been able to find a way to use the webAPI for stable scraping. So I made some changes and the script is now using the mobile app API. I scraped some large accounts without any issues. Please check out https://github.com/Gertje823/Vinted-Scraper/tree/47-depop-403 and let me know if it is also stable for you guys so I can push this to the main branch.

@Gertje823 Gertje823 linked a pull request May 31, 2024 that will close this issue
@Relax594
Copy link

@Gertje823 works perfectly, thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants