Cookies are not working

[abd-falyoun]

I've logged in into a webiste using puppteer then I saved cookies to a file, after that I provided these cookies to website-scraper

// Parsing cookies from file

const cookies = ...;

scrape({
  urls: ['http://example.com/'],
  directory: '/path/to/save',
  request: {
    headers: {
      'User-Agent': 'Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 4 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19',
      
      'Set-Cookie': arrayOfCookies, // Not working
      'cookie': arrayOfCookes // Not working
    }
  }
});

When I run the above code it redirects me to login page.

[s0ph1e]

Hi @abd-falyoun

From your options it's hard to understand what went wrong without steps to reproduce. Maybe your website requires some additional headers or does another checks to reject access for automated tools.

To verify it works you can try to call request directly with your headers and check the response.

Also, I'm not sure what is the type of variable arrayOfCookies in your example (according to name it's an array), but header should have string value. Please double check your headers options.

[no-response]

This issue has been automatically closed because there has been no response from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[rjenk22]

How can I convert json cookie to format needed for Set-Cookie?

Hi @abd-falyoun

From your options it's hard to understand what went wrong without steps to reproduce. Maybe your website requires some additional headers or does another checks to reject access for automated tools.

To verify it works you can try to call request directly with your headers and check the response.

Also, I'm not sure what is the type of variable arrayOfCookies in your example (according to name it's an array), but header should have string value. Please double check your headers options.

How can I convert json cookie to format needed for Set-Cookie?

[rjenk22]

If I have 6 cookies like this:
https://i.imgur.com/QTQl9Jx.png
Do I need to set 'Set-Cookie' six times? Because I am trying to set cookies to log into a site then scrape it, however the scraped output keeps outputting the login page. Is there a way to convert puppeteer json cookies to cookies I can set here?

[s0ph1e]

Hi @Christian-Martinez1

The Set-Cookie header is sent by the server in response to an HTTP request, which is used to create a cookie on the user's system. The Cookie header is included by the client application with an HTTP request sent to a server, if there is a cookie that has a matching domain and path.

You may need to include cookie header for the request, and not set-cookie, please double check your headers.
You can use log and debug to see what headers are sent by scraper.

[cecilemuller]

In case someone is still looking for the answer, you can use request.cookieJar to set the matching got option:

• https://github.com/website-scraper/node-website-scraper#request
• https://github.com/sindresorhus/got/blob/main/documentation/2-options.md#cookiejar

Example:

import {CookieJar} from "tough-cookie";
import scrape from "website-scraper";

const cookieJar = new CookieJar();
await cookieJar.setCookie("foo=bar", "https://example.com");

const result = await scrape({
  urls: ["https://example.com"],
  directory: "/path/to/save/",
  request: {
    cookieJar
  }
});