How to import URLS?

[dynabler]

How do I import URLS from an external file? I have seen readFile #10. Is this available? If so, how do I use it?

[philippta]

We have an example showing you can do to that:

script:

https://github.com/philippta/flyscrape/blob/master/examples/urls_from_file.js

import urls from "./urls.txt"

export const config = {
  urls: urls.split("\n")
};

export default function({ doc }) {
  return {
    title: doc.find("title").text().trim(),
  };
}

file with urls:

https://github.com/philippta/flyscrape/blob/master/examples/urls.txt

https://news.ycombinator.com/newest
https://news.ycombinator.com/ask
https://news.ycombinator.com/show

[dynabler]

In my case, it grabs only one URL from ./url.txt. url.txt has 1000 URLs. Not sure why, I copy & paste exactly as the examples. Any ideas?

[philippta]

Without being able to reproduce the issue, my best guess would be the difference in line terminators between windows (\r\n) and linux, macos (\n).

Try changing your config to this and see if it works:

import urls from "./urls.txt"

export const config = {
  urls: urls.replaceAll("\r\n", "\n").split("\n").filter(Boolean)
};

[philippta]

Update

With the v07.2 release, this got a lot more user friendly.
Flyscrape now automatically cleans up the urls and also ignores empty lines in the file.
This also means, the JavaScript can be simplified a lot.

import urls from "./urls.txt"

export const config = {
  urls: urls.split("\n")
};