Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dataset as requestsFromUrl #2297

Open
apify-alexey opened this issue Jan 22, 2024 · 0 comments
Open

dataset as requestsFromUrl #2297

apify-alexey opened this issue Jan 22, 2024 · 0 comments
Labels
feature Issues that represent new features or improvements to existing features.

Comments

@apify-alexey
Copy link

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/core

Feature

Currently when dataset used as requestsFromUrl and contains request-alike data item objects userData is ignored, instead when-if userData contains urls each added as separate request.
Sample https://api.apify.com/v2/datasets/rl2e6Xcs11vMMCvyN/items?clean=true&format=json

[{
  "url": "https://www.facebook.com/wayoflifesq/",
  "userData": {
    "channelDescriptionLinks": [
      {
        "text": "Instagram",
        "url": "https://www.instagram.com/wayoflifesq/?hl=en"
      },
      {
        "text": "Twitter",
        "url": "https://twitter.com/wayoflifesq?lang=en"
      },
      {
        "text": "Facebook",
        "url": "https://www.facebook.com/wayoflifesq/"
      }
    ],
    "inputChannelUrl": "https://www.youtube.com/channel/UC3kfx81WYXgNBf1bNIcO5Fg",
    "channelName": "WAYOFLIFESQ"
  }
}]

Motivation

Its almost working https://console.apify.com/view/runs/S4KFp0kfbV3NEXwAJ
Idea is to pass dataset as startUrls in input:

{
  "startUrls": [
    {
      "requestsFromUrl": "https://api.apify.com/v2/datasets/rl2e6Xcs11vMMCvyN/items?clean=true&format=csv"
    }
  ]
}

Ideal solution or implementation, and any additional constraints

Just add proper parsing, needs to be more accurate

Alternative solutions or implementations

No response

Other context

No response

@apify-alexey apify-alexey added the feature Issues that represent new features or improvements to existing features. label Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Issues that represent new features or improvements to existing features.
Projects
None yet
Development

No branches or pull requests

1 participant