Skip to content

Commit

Permalink
Stop failing the Freshmaker scraper when the connection times out unt…
Browse files Browse the repository at this point in the history
…il FACTORY-3955 is resolved

This will keep the scraper OpenShift cron job from being marked as failed all the time.
  • Loading branch information
mprahl committed Jun 5, 2019
1 parent 4e69b2b commit f8850f9
Showing 1 changed file with 11 additions and 1 deletion.
12 changes: 11 additions & 1 deletion scrapers/freshmaker.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import xml.etree.ElementTree as ET

import neomodel
from requests.exceptions import ConnectionError

from scrapers.base import BaseScraper
from estuary.models.freshmaker import FreshmakerEvent, FreshmakerBuild
Expand Down Expand Up @@ -44,7 +45,16 @@ def query_api_and_update_neo4j(self):
fm_url = self.freshmaker_url
while True:
log.debug('Querying {0}'.format(fm_url))
rv_json = session.get(fm_url, timeout=60).json()
try:
rv_json = session.get(fm_url, timeout=60).json()
except ConnectionError:
# TODO: Remove this once FACTORY-3955 is resolved
log.error(
'The connection to Freshmaker at %s failed. Skipping the rest of the scraper.',
fm_url,
)
break

for fm_event in rv_json['items']:
try:
int(fm_event['search_key'])
Expand Down

0 comments on commit f8850f9

Please sign in to comment.