Cannot download CSV/Media file using `DfareportingService#get_file` #290

rafaelsales · 2015-09-30T03:14:44Z

Issues: Option download_dest in DfareportingService#get_file does not work.

Testing file:

require 'google/apis/dfareporting_v2_1'
require 'hurley'

def auth
  @auth ||= begin
    scope = 'https://www.googleapis.com/auth/dfareporting'
    DoubleClick::Client.new(scope).authorization # DoubleClick::Client is my helper class for auth
  end
end

def service
  @service ||= Google::Apis::DfareportingV2_1::DfareportingService.new.tap do |service|
    service.authorization = auth
  end
end

def get_random_report_file(profile_id:, report_id:)
  service.list_report_files(profile_id, report_id).items.first do |file|
    file.status == 'REPORT_AVAILABLE'
  end
end

def download_ackward(file_metadata)
  file_get = Hurley.get(file_metadata.urls.api_url) do |request|
    request.header['Authorization'] = "Bearer #{auth.access_token}"
  end
  file_get.body
end

def download_beatiful(file_metadata)
  file_dest = '/tmp/report_beautiful.csv'
  service.get_file(file_metadata.report_id, file_metadata.id, download_dest: file_dest)
  File.new(file_dest).read
end

file_metadata = get_random_report_file(profile_id: 123, report_id: 456)

puts "--- With Ackward download method:"
puts download_ackward(file_metadata).lines[0..9]

puts "\n"
puts "--- With Beatiful download method:"
puts download_beatiful(file_metadata).lines[0..9]

Outcome:

--- With Ackward download method:
my-report

Date/Time Generated,9/29/15 4:48 AM
Report Time Zone,America/Los_Angeles
SubAccount Name,MyAccountName
Account ID,12345
Date Range,9/28/15 - 9/28/15

Report Fields
Campaign,Site (DCM),Date,Campaign End Date,Campaign Start Date,Campaign ID,Site ID (DCM),State/Region,Country,Placement,Placement ID,Site (Site Directory),Site ID (Site Directory),Advertiser,Advertiser ID,Impressions,Clicks,Click Rate,Cost Per Click,Effective CPM

--- With Beatiful download method:
Writing chunk (632 bytes)
Success - nil

<HTML>
<HEAD>
<TITLE>Temporary Redirect</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Temporary Redirect</H1>
The document has moved <A HREF="https://storage.googleapis.com/dfa_-my-report....csv?GoogleAccessId=abc@developer.gserviceaccount.com&amp;Expires=123&amp;Signature=xyz">here</A>.
</BODY>
</HTML>

Expectation: The get_file(..., download_dest: ...) should have downloaded the file correctly, instead if returns that "Temporary Redirect" page.

The text was updated successfully, but these errors were encountered:

sqrrrl · 2015-09-30T19:21:36Z

Will take a look at. Would be helpful if you can capture the raw HTTP response from the download request (e.g. using curl or whatever) since I don't have a DFA account I can use to test. Just need to verify that they're sending a proper redirect.

FYI, you can simplify the workaround a little - there's a generic http() method on the service that will
add the auth header and also supports writing directly to files:

service.http(:get, file_metadata.urls.api_url, download_dest: '/tmp/report.csv')

sqrrrl · 2015-10-02T23:16:58Z

Don't need the HTTP response. Was able to reproduce and am looking at a fix. Might be a little while though since the preferred fix involves some fixes to Hurley (http library)

rafaelsales · 2015-10-04T23:08:46Z

Thank you!

bruzos · 2015-10-05T13:57:21Z

@rafaelsales did you use google services accounts to connect to the DCM webservice ?.

I need to work with that webservice but I have problems about permisions when my script connects with that Webservice using google service account.

Are you using that flow connection ?

Thank you

bruzos · 2015-10-08T09:21:28Z

I can confirm google account service doesn´t works with DCM if the google Apps Domain is not setup.

rafaelsales · 2015-10-08T22:29:59Z

I couldn't access DFA reporting service with service account. Had to use client id, client secret and manually create a first refresh token.

rafaelsales · 2015-10-08T22:30:57Z

btw, I tried with my company google apps

bruzos · 2015-10-09T08:25:13Z

Rafaelsales thank you for your reply.

I could make it work using installed flow. there is no needed to use google apps with this authentication flow, this way is recommended by google.

rafaelsales · 2015-10-09T18:14:49Z

@bruzos when you say "it work" you mean the get_file method or just the auth? Cause in theory web and installed flow are not really different from each other: https://github.com/google/google-api-ruby-client/blob/669aaf01a401628c05c9081789ba4204919c6ac3/lib/google/api_client/client_secrets.rb#L29-L47
It's just that installed flow is little bit more convenient for mobile apps.

bruzos · 2015-10-13T08:38:04Z

Hi @rafaelsales I have tried your code with the same behavior you wrote. I think the problem is in the Hurley client it doesn't handle redirections.
Your are right, but google recommend as alternative to google service account use this flow, but if you manage by your self the access token life, I think it is ok

rafaelsales · 2015-10-13T17:11:17Z

@bruzos I'm pretty sure Hurley actually does follow redirects: https://github.com/lostisland/hurley/blob/9be84b325c9a396a64268cd7bbca94080d9485ee/test/client_test.rb#L319

See the method download_ackward in the PR description - I use Hurley to download the CSV and it works.
In my opinion it's the google-api-ruby-client that is using the wrong URL to download the CSV, because I could make that work using file_metadata.urls.api_url.

sqrrrl · 2015-10-13T19:01:57Z

It's a combination of api client + hurley bugs.

The issues with the client is it isn't expecting a redirect for downloads. APIs aren't supposed to redirect here, but a few do and the client should handle it. The problem is that Hurley is broken when it comes to streaming responses and redirects. There is a mechanism to filter streaming responses by status code, but it doesn't work. There's also a 2nd issue that it loses track of the fact that the caller wants to stream when a redirect happens. There are pending PRs (lostisland/hurley#28) and lostisland/hurley#34)) for both issues that I hope will be in the next release of Hurley. Once those are fixed I'll update the client to handle the redirect properly.

bruzos · 2015-10-14T09:26:03Z

@sqrrrl and @rafaelsales thank you very much. I will patiently wait for it :)

rafaelsales · 2015-10-14T10:43:37Z

@sqrrrl thanks a lot for the explanation!

bruzos · 2015-12-03T10:29:48Z

Hi @sqrrrl it seem the pending PRs for Hurley client we was waiting for have been merged to master.

… on 20x response. Includes temporary patch to Hurley until 0.3 released

sqrrrl · 2015-12-16T20:23:55Z

They have been, but still not released so I can do a proper dependency on them. That said, I did hack in a fix for the next release which I'll push shortly.

sqrrrl added this to the 0.9 milestone Sep 30, 2015

sqrrrl added a commit that referenced this issue Dec 16, 2015

Issue #290 - Fix redirects during downloads, only stream body content…

a1b5f6c

… on 20x response. Includes temporary patch to Hurley until 0.3 released

sqrrrl closed this as completed Dec 16, 2015

yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Apr 6, 2020

JustinBeckwith assigned sqrrrl Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot download CSV/Media file using `DfareportingService#get_file` #290

Cannot download CSV/Media file using `DfareportingService#get_file` #290

rafaelsales commented Sep 30, 2015

sqrrrl commented Sep 30, 2015

sqrrrl commented Oct 2, 2015

rafaelsales commented Oct 4, 2015

bruzos commented Oct 5, 2015

bruzos commented Oct 8, 2015

rafaelsales commented Oct 8, 2015

rafaelsales commented Oct 8, 2015

bruzos commented Oct 9, 2015

rafaelsales commented Oct 9, 2015

bruzos commented Oct 13, 2015

rafaelsales commented Oct 13, 2015

sqrrrl commented Oct 13, 2015

bruzos commented Oct 14, 2015

rafaelsales commented Oct 14, 2015

bruzos commented Dec 3, 2015

sqrrrl commented Dec 16, 2015

Cannot download CSV/Media file using DfareportingService#get_file #290

Cannot download CSV/Media file using DfareportingService#get_file #290

Comments

rafaelsales commented Sep 30, 2015

sqrrrl commented Sep 30, 2015

sqrrrl commented Oct 2, 2015

rafaelsales commented Oct 4, 2015

bruzos commented Oct 5, 2015

bruzos commented Oct 8, 2015

rafaelsales commented Oct 8, 2015

rafaelsales commented Oct 8, 2015

bruzos commented Oct 9, 2015

rafaelsales commented Oct 9, 2015

bruzos commented Oct 13, 2015

rafaelsales commented Oct 13, 2015

sqrrrl commented Oct 13, 2015

bruzos commented Oct 14, 2015

rafaelsales commented Oct 14, 2015

bruzos commented Dec 3, 2015

sqrrrl commented Dec 16, 2015

Cannot download CSV/Media file using `DfareportingService#get_file` #290

Cannot download CSV/Media file using `DfareportingService#get_file` #290