Skip to content
This repository has been archived by the owner on Apr 9, 2020. It is now read-only.

Commit

Permalink
Production log was flooded with 404 scraping errors, reduce to debug …
Browse files Browse the repository at this point in the history
…level
  • Loading branch information
Nagasaki45 committed Jul 25, 2016
1 parent 81f4412 commit bfb890d
Showing 1 changed file with 3 additions and 6 deletions.
9 changes: 3 additions & 6 deletions lib/pipeline/stats_scraper.ex
Expand Up @@ -89,20 +89,17 @@ defmodule Krihelinator.Pipeline.StatsScraper do
end

@doc """
When HTTPoison fails, log the failure in the most indicative way.
Several failures are ignorable. Log them for debugging and let the process
crash otherwise.
"""
def handle_failure(httpoison_response, repo_name) do
msg = case httpoison_response do
{:ok, %{status_code: 404}} ->
"Page not found (404)"
{:ok, %{status_code: 451}} ->
"Repository unavailable due to DMCA takedown"
{:ok, %{body: body, status_code: status_code}} ->
"Status code: #{status_code}\n#{Floki.text(body)}"
{:error, httpoison_error} ->
inspect(httpoison_error)
end
Logger.warn "Failed to scrape #{repo_name}: #{msg}"
Logger.debug "Failed to scrape #{repo_name}: #{msg}"
end

end

0 comments on commit bfb890d

Please sign in to comment.