INFO: How to scrape a API using POST method calls

[ahsandar]

How to scrape a API using POST method calls, are there any examples or docs. Any sample code would be nice. Thanks

[Ziinc]

You can create a custom fetcher that uses the Poison.post function and then override settings in the spider.

[ahsandar]

Thanks for the response as I am new to Elixir and never used crawly would you happen to have a sample. I can there are plenty of samples around scrapping website but none covering this use case.

[Ziinc]

Fetcher docs: https://hexdocs.pm/crawly/Crawly.Fetchers.Fetcher.html#content

Example fetcher: https://github.com/oltarasenko/crawly/blob/master/lib/crawly/fetchers/httpoison_fetcher.ex

override_settings callback: https://hexdocs.pm/crawly/Crawly.Spider.html#c:override_settings/0

override_settings example: https://github.com/oltarasenko/crawly/blob/7bf04b0f23e02c9bc7c5b12dc962efb25ca88371/test/settings_test.exs#L47

[oltarasenko]

@Ziinc thanks for helping!

@ahsandar But why do you need to use POST requests? Otherwise, if it's about fetching ajax parts of your page, you could try to use the browser rendering approach!

[ahsandar]

@oltarasenko i am looking for fetching data from APIs not in websites which could be using POST. As I want to use same tool to consistently use to collect data from websites and APIs and Crawly looks to be a great tool to achieve this

[oltarasenko]

I would assume this is resolved now. Please re-open if it's not the case!