403 error on udemy domain

[JuicyLung91]

Seems like udemy.com is blocking every fetch and returns a 403 status. Already tried various useragents. Like: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12

Any idea how to crawl udemy pages?

The issue also appears on the demo page.

[JuicyLung91]

Allow cookies for the CurlDispatcher by adding CURLOPT_COOKIEJAR solves the problem.
Just added:

new CurlDispatcher([
/*...*/
            CURLOPT_COOKIEJAR => './cookie.txt', 
/*...*/
]);

[oscarotero]

That's weird, because in theory, cookies are enabled automatically: https://github.com/oscarotero/Embed/blob/master/src/Http/CurlDispatcher.php#L49

[JuicyLung91]

Are you sure with the path for CURLOPT_COOKIEJAR:
https://github.com/oscarotero/Embed/blob/a20d711d573db750a1a65a5ff61947749183c49e/src/Http/CurlDispatcher.php#L50 ?

According to this https://www.php.net/manual/en/book.curl.php#90821
The CURLOPT_COOKIEJAR needs a path like this: dirname(FILE).'/cookie.txt';

[oscarotero]

That creates the cookie file in the same directory of this class, instead in the temp directory, so it wont be deleted automatically. I don't think this affects.
Could it be because txt extension is required?

[JuicyLung91]

Just tried it out on my local setup:
$cookies = str_replace('//', '/', sys_get_temp_dir().'/embed-cookies.'.uniqid().'.text');
it works.

So yes maybe the .txt extension is needed.

[oscarotero]

ok, thanks!