Timeouts in getReportDownloadUrlWithOptions

[kerkela]

This looks a lot like #20 which @nicholaschen indicated should have been fixed by the Suds client having a 1 hour timeout,
but I'm seeing this timeout after about 90 seconds. It's happening consistently for certain (larger) accounts with certain (larger) sets of dimensions.
Meanwhile it succeeds for smaller accounts and/or fewer dimensions.
FWIW I'm using googleads 5.3.0 & DFP version v201702, though it failed identically for slightly older versions of each on testing.
I've doublechecked that socket.setdefaulttimeout has no impact as well.

Stack trace:
sb1.tests.e2e: ERROR: Problem downloading 20170410x225736xC4C907A4 filename=download_util.py, lineno=75
Traceback (most recent call last):
...
File "/Users/stacy/distillerizer/sb1/sources/dfp_util.py", line 506, in get_report_url
return service.getReportDownloadUrlWithOptions(job_id, options)
File "/Users/stacy/distillerizer/venv/lib/python2.7/site-packages/googleads/common.py", line 643, in MakeSoapRequest
for arg in args])
File "/Users/stacy/distillerizer/venv/lib/python2.7/site-packages/suds/client.py", line 521, in call
return client.invoke(args, kwargs)
File "/Users/stacy/distillerizer/venv/lib/python2.7/site-packages/suds/client.py", line 581, in invoke
result = self.send(soapenv)
File "/Users/stacy/distillerizer/venv/lib/python2.7/site-packages/suds/client.py", line 613, in send
reply = self.options.transport.send(request)
File "/Users/stacy/distillerizer/venv/lib/python2.7/site-packages/suds/transport/http.py", line 82, in send
fp = self.u2open(u2request)
File "/Users/stacy/distillerizer/venv/lib/python2.7/site-packages/suds/transport/http.py", line 132, in u2open
return url.open(u2request, timeout=tm)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1200, in do_open
r = h.getresponse(buffering=True)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1132, in getresponse
response.begin()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 734, in recv
return self.read(buflen)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 621, in read
v = self._sslobj.read(len or 1024)
SSLError: ('The read operation timed out',)

My code:
downloader = self._dfp_client.GetDataDownloader(version=VERSION)
job_id = downloader.WaitForReport(report_job)
service = self._dfp_client.GetService('ReportService', version=VERSION)
report_url = get_report_url(job_id, service)
response = urllib2.urlopen(report_url)
while True:
chunk = response.read(_CHUNK_SIZE)
if not chunk:
break
file_handle.write(chunk)

report job:
{
reportQuery =
(ReportQuery){
dimensions[] =
"DATE",
"LINE_ITEM_ID",
"LINE_ITEM_NAME",
"LINE_ITEM_TYPE",
"ORDER_ID",
"ORDER_NAME",
"AD_UNIT_ID",
"AD_UNIT_NAME",
"CREATIVE_SIZE",
"DEVICE_CATEGORY_ID",
"DEVICE_CATEGORY_NAME",
"COUNTRY_NAME",
"COUNTRY_CRITERIA_ID",
"ADVERTISER_ID",
"ADVERTISER_NAME",
"CREATIVE_NAME",
"CREATIVE_ID",
"CREATIVE_TYPE",
"CREATIVE_BILLING_TYPE",
"PLACEMENT_NAME",
"PLACEMENT_ID",
adUnitView = "FLAT"
columns[] =
"AD_SERVER_IMPRESSIONS",
"AD_SERVER_CLICKS",
"AD_EXCHANGE_LINE_ITEM_LEVEL_IMPRESSIONS",
"AD_EXCHANGE_LINE_ITEM_LEVEL_CLICKS",
"AD_SERVER_WITHOUT_CPD_AVERAGE_ECPM",
dimensionAttributes[] =
"AD_UNIT_CODE",
"LINE_ITEM_COST_PER_UNIT",
startDate =
(Date){
year = 2017
month = 4
day = 9
}
endDate =
(Date){
year = 2017
month = 4
day = 9
}
dateRangeType = "CUSTOM_DATE"
useSalesLocalTimeZone = False
}
}

with the following options
{'exportFormat': 'CSV_DUMP', 'useGzipCompression': True}
or
{'exportFormat': 'CSV_DUMP', 'useGzipCompression': False}

[msaniscalchi]

Thanks for the report, we'll look into it.

[manoochehri]

Hi @grivescorbett - any updates on this bug? This is causing a blocking issue for several large DFP customers.

[grivescorbett]

Hi Michael, I have a fix for this. I've left the office for the weekend but on Monday I will figure out if we want to do an out of band release or if not will just post a patch here. Best, Gabe

On Fri, Apr 21, 2017 at 5:52 PM Michael Manoochehri < ***@***.***> wrote: Hi @grivescorbett <https://github.com/grivescorbett> - any updates on this bug? This is causing a blocking issue for several large DFP customers. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#200 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYJjAkVS5vv9ts4RDk2sEejvsJy3Iekks5rySUWgaJpZM4M5caM> .

-- Gabe Rives-Corbett | Developer Programs Engineer, DFP API | grivescorbett@google.com

[manoochehri]

Thanks @grivescorbett - I appreciate it, looking forward to the fix.

[manoochehri]

@grivescorbett - any update on your fix? Even a patch would be helpful for us.

[grivescorbett]

Hi Michael,

Sorry for the delay. I'm targeting a new release of the python client lib by EOD Eastern today.

Best,
Gabe

[grivescorbett]

Resolved in 5.4.0