Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting time out error with 'delete' request method #1658

Closed
1 task
LeMoussel opened this issue Nov 7, 2022 · 0 comments · Fixed by #2039
Closed
1 task

Getting time out error with 'delete' request method #1658

LeMoussel opened this issue Nov 7, 2022 · 0 comments · Fixed by #2039
Labels
bug Something isn't working.

Comments

@LeMoussel
Copy link

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/http (HttpCrawler)

Issue description

Setting delete method for request doesn't work. Request fail with time out.

curl equivalent
curl -X DELETE "http://httpbin.org/delete" -H "accept: application/json"

Code sample

import { HttpCrawler } from 'crawlee';

const http_crawler = new HttpCrawler({
    maxRequestRetries: 1,
    navigationTimeoutSecs: 5,
    additionalMimeTypes: ['application/json'],

    requestHandler: async (ctx) => {
        ctx.log.info(`Processing ${ctx.request.url}...`);
        ctx.log.info(`Response body: ${ctx.body}`);
        //console.dir(ctx.json);
    },
    failedRequestHandler: async (ctx) => {
        ctx.log.error(`Request ${ctx.request.url} failed too many times. JSON Response body not scraped!`);
    }
});

const delete_request  = [
    {
        url:'http://httpbin.org/delete',
        method: 'DELETE',
        headers: {'accept':'application/json'},
    },
];
await http_crawler.run(delete_request);

Package version

3.0.4

Node.js version

v16.17.1

Operating system

Windows

Apify platform

  • Tick me if you encountered this issue on the Apify platform

Priority this issue should have

High (immediate attention needed)

I have tested this on the next release

No response

Other context

No response

@LeMoussel LeMoussel added the bug Something isn't working. label Nov 7, 2022
HamzaAlwan added a commit to HamzaAlwan/crawlee that referenced this issue Aug 21, 2023
This commit addresses an issue in the @crawlee/http (HttpCrawler) and @crawlee/cheerio (CheerioCrawler) packages related to setting the DELETE method for requests. The problem caused requests to fail with a timeout, rendering the functionality unusable.

Closes [apify#1658](apify#1658)
HamzaAlwan added a commit to HamzaAlwan/crawlee that referenced this issue Aug 21, 2023
This commit addresses an issue in the @crawlee/http (HttpCrawler) and @crawlee/cheerio (CheerioCrawler) packages related to setting the DELETE method for requests. The problem caused requests to fail with a timeout, rendering the functionality unusable.

Closes [apify#1658](apify#1658)
B4nan pushed a commit that referenced this issue Aug 21, 2023
This commit addresses an issue in the @crawlee/http (HttpCrawler) and
@crawlee/cheerio (CheerioCrawler) packages related to setting the DELETE
method for requests. The problem caused requests to fail with a timeout,
rendering the functionality unusable.

Closes #1658
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant