How to write to file the results?

[ivanhercaz]

Hi Crawly Community!
Thank you a lot for this package that it is really useful. I follow the example given on the README of the repository and it works really fine, but something is wrong when writting to a file but I am not getting any error.

I have the next configuration:

config :crawly,
  closespider_timeout: 10,
  concurrent_requests_per_domain: 8,
  closespider_itemcount: 100,
  middlewares: [
    Crawly.Middlewares.DomainFilter,
    Crawly.Middlewares.UniqueRequest,
    {Crawly.Middlewares.UserAgent, user_agents: ["Crawly Bot"]}
  ],
  pipelines: [
    Crawly.Pipelines.JSONEncoder,
    {Crawly.Pipelines.WriteToFile, extension: "jl", folder: "/tmp"}
  ]

And what I get when I run iex -S mix run -e "Crawly.Engine.start_spider(CrawlyCabildogc.CabildoNews)" is:

15:45:07.723 [debug] Starting data storage
15:45:07.729 [debug] Starting the manager for CrawlyCabildogc.CabildoNews
15:45:07.732 [debug] Starting requests storage worker for CrawlyCabildogc.CabildoNews...
15:45:07.733 [debug] Started 4 workers for CrawlyCabildogc.CabildoNews
Interactive Elixir (1.14.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> 
15:45:07.772 [warning] Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'

15:45:09.625 [debug] Stored item: %{title: "\n                                    El vicepresidente del Cabildo anuncia la puesta en marcha de una nueva línea directa de transporte interurbano que unirá el Tablero de Maspalomas y el Campus Universitario\n                                "}

Of course, with more than one result. I think the only warning I am getting is not the problem.

The source code of the spider is practically the same than the one in the example you use on the README, with a few changes to adapt it to my case. I share here what I have on parse_item/1:

  @impl Crawly.Spider
  def parse_item(response) do
    {:ok, document} = Floki.parse_document(response.body)

    items =
      document
      |> Floki.find(".asset-abstract")
      |> Enum.map(fn x ->
        %{
          title: Floki.find(x, ".content .asset-title a")
          |> Floki.text()
        }
      end)

    next_requests =
      document
      |> Floki.find(".pagination li:last-child.page-item a")
      |> Floki.attribute("href")
      |> Enum.map(fn url ->
        Crawly.Utils.build_absolute_url(url, response.request.url)
        |> Crawly.Utils.request_from_url()
      end)

    %{items: items, requests: next_requests}
  end

Any idea about why I am not obtaining the results written on a file?

Thank you in advance for your attention.

Regards,
Iván

Edit: according to Crawly.Pipelines.WriteToFile docs, if the folder doesn't exist, it is created:

:folder, optional. The folder in which the file will be created. Defaults to current project's folder. If provided folder does not exist it's created.

I test now using {Crawly.Pipelines.WriteToFile, extension: "jl", folder: "/tmp/crawly"} and I checked that the folder crawly is not created.

[dogweather]

Here's my working pipelines section:

  pipelines: [
    Crawly.Pipelines.JSONEncoder,
    {Crawly.Pipelines.WriteToFile, extension: "jsonl", folder: "./tmp"}
  ],

Maybe it's my use of JSONEncoder? Also, I'm using a pre-created local tmp folder. (IMO that makes a lot more sense than fishing around in /tmp.)

[ivanhercaz]

Here's my working pipelines section:

  pipelines: [
    Crawly.Pipelines.JSONEncoder,
    {Crawly.Pipelines.WriteToFile, extension: "jsonl", folder: "./tmp"}
  ],

Maybe it's my use of JSONEncoder? Also, I'm using a pre-created local tmp folder. (IMO that makes a lot more sense than fishing around in /tmp.)

Hi @dogweather! Thank you for your reply.

I already tried with a pre-created local tmp folder but it doesn't work. The main difference with your pipelines is the extension: I use jl as in the README's example and you use jsonl, but nothing changes if I use one or the other.

I don't know what you mean with your use of JSONEncoder. I already have it in the pipelines, but maybe I something I don't understand.

The main problem here is that I am not getting any error or warning related with the creation of the file.

What is the behavior of this pipeline? When the data is crawled it is added progressively? Or is it created when the crawler "finish"?

[Ziinc]

The data is added as more parsed items are received. They will be appended to the file.
It is possible that your program does not have sufficient permissions to write to and append to the spider's output file, that would be the first thing I would check through iex that the File module's apis work as expected.

[ivanhercaz]

The data is added as more parsed items are received. They will be appended to the file. It is possible that your program does not have sufficient permissions to write to and append to the spider's output file, that would be the first thing I would check through iex that the File module's apis work as expected.

Hi, @Ziinc! Thank you for your answer. I thought about a possible issue with the permissions to write through iex, but I check on these days. Now I replicated again to show you the result:

iex(1)> File.write!("./tmp/test.jl", "text test")
:ok
iex(2)> File.ls "./tmp"
{:ok, ["test.jl"]}

I checked the ./tmp folder and the file is correctly created with the specified content. Then I run the spider to check again and nothing is written on a file 😢 .

[dogweather]

I don't know what you mean with your use of JSONEncoder. I already have it in the pipelines, but maybe I something I don't understand.

Oh, duh. You're totally right. I'm honestly not sure about when data is added, but I'm working on the code today and I'll see.

My code is open source, btw. You can try it out and see the full configuration. https://github.com/public-law/oregon-revised-statutes-crawler

[ivanhercaz]

Thank you, @dogweather! I will check your code as soon as possible.