Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crawlData 的请求参数传递有问题 #30

Closed
ConteMan opened this issue May 3, 2023 · 4 comments · Fixed by #31
Closed

crawlData 的请求参数传递有问题 #30

ConteMan opened this issue May 3, 2023 · 4 comments · Fixed by #31
Labels
bug Something isn't working

Comments

@ConteMan
Copy link

ConteMan commented May 3, 2023

Bug 预期

crawlData 请求数据,使用 CrawlDataDetailTargetConfigparams 传递请求参数,发现实际请求并未携带参数。

查看源码发现 此处params 参数处理为 search,然后在 此处 传递给 https.request() 方法进行请求。

查看 Node.js 文档,http.request() 的解释 HTTP #httprequestoptions-callback | Node.js v18.16.0 Documentation,请求参数应该是拼接在 path 后。

最小可重复的例子

const pageSize = 30;
    const sort = 'time';
    const rating = 'all';
    const filter = 'all';
    const mode = 'list';
    const url = `https://${type}.${this.BASE_URL}/people/${uid}/${status}`;
    const request: CrawlDataDetailTargetConfig = {
      url,
      method: "GET",
      params: {
        start: (page - 1) * pageSize,
        sort,
        rating,
        filter,
        mode,
      },
      data: {},
      timeout: 30000,
      maxRetry: 0
    };

    const crawler = Crawl({
      enableRandomFingerprint: true
    });
    const res = await crawler.crawlData(request);

报错信息

无报错

x-crawl 版本

7.0.0

Node 版本

18.15.0

包管理器

npm

包管理器版本

8.3.1

@ConteMan ConteMan added the bug Something isn't working label May 3, 2023
@github-actions
Copy link

github-actions bot commented May 3, 2023

Message that will be displayed on users' first issue

@coder-hxl
Copy link
Owner

coder-hxl commented May 3, 2023

感谢您的 Bug 反馈,我到时候开机再处理,现在挺晚的。

@ConteMan
Copy link
Author

ConteMan commented May 3, 2023

感谢您的 Bug 反馈,我到时候开机再处理,现在挺晚的。 httphttps 这两个不能传 RequestOptions 交叉 URL 类型,URL 类型有 search ,但 RequestOptions 没有。

你合适时间处理就好,我的作息有点异常 😄。

@coder-hxl
Copy link
Owner

已经修复并发布了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants