Issue with download reports and custom reporting settings

[chiragvels]

Hello All,

I am not able to download report with this new API.
it was issue with makeReportRequest under ReportDownloader.php
I am not able to get $response,
beyond this line not able go.
$response = $this->httpClient->request(
'POST',
rtrim($this->session->getEndpoint(), '/')
. self::$REPORT_DOWNLOAD_URL_PATH,
$requestOptions
);

I am not able to make request().

Also not going to ClientException $e, so that I can understood what can be problem.

Please help me here, how to proceed now? Neither getting response nor exception.

guzzle http client
GuzzleHttp\Client Object
(
[config:GuzzleHttp\Client:private] => Array
(
[handler] => GuzzleHttp\HandlerStack Object
(
[handler:GuzzleHttp\HandlerStack:private] => Closure Object
(
[static] => Array
(
[default] => Closure Object
(
[static] => Array
(
[default] => GuzzleHttp\Handler\CurlMultiHandler Object
(
[factory:GuzzleHttp\Handler\CurlMultiHandler:private] => GuzzleHttp\Handler\CurlFactory Object
(
[handles:GuzzleHttp\Handler\CurlFactory:private] =>
[maxHandles:GuzzleHttp\Handler\CurlFactory:private] => 50
)

                                                        [selectTimeout:GuzzleHttp\Handler\CurlMultiHandler:private] => 1
                                                        [active:GuzzleHttp\Handler\CurlMultiHandler:private] => 
                                                        [handles:GuzzleHttp\Handler\CurlMultiHandler:private] => Array
                                                            (
                                                            )

                                                        [delays:GuzzleHttp\Handler\CurlMultiHandler:private] => Array
                                                            (
                                                            )

                                                    )

                                                [sync] => GuzzleHttp\Handler\CurlHandler Object
                                                    (
                                                        [factory:GuzzleHttp\Handler\CurlHandler:private] => GuzzleHttp\Handler\CurlFactory Object
                                                            (
                                                                [handles:GuzzleHttp\Handler\CurlFactory:private] => 
                                                                [maxHandles:GuzzleHttp\Handler\CurlFactory:private] => 3
                                                            )

                                                    )

                                            )

                                        [parameter] => Array
                                            (
                                                [$request] => 
                                                [$options] => 
                                            )

                                    )

                                [streaming] => GuzzleHttp\Handler\StreamHandler Object
                                    (
                                        [lastHeaders:GuzzleHttp\Handler\StreamHandler:private] => Array
                                            (
                                            )

                                    )

                            )

                        [parameter] => Array
                            (
                                [$request] => 
                                [$options] => 
                            )

                    )

                [stack:GuzzleHttp\HandlerStack:private] => Array
                    (
                        [0] => Array
                            (
                                [0] => Closure Object
                                    (
                                        [static] => Array
                                            (
                                                [logger] => Monolog\Logger Object
                                                    (
                                                        [name:protected] => AW_REPORT_DOWNLOADER
                                                        [handlers:protected] => Array
                                                            (
                                                                [0] => Monolog\Handler\StreamHandler Object
                                                                    (
                                                                        [stream:protected] => 
                                                                        [url:protected] => /var/www/html/aw_logs/logs/report-downloader.log
                                                                        [errorMessage:Monolog\Handler\StreamHandler:private] => 
                                                                        [filePermission:protected] => 
                                                                        [useLocking:protected] => 
                                                                        [dirCreated:Monolog\Handler\StreamHandler:private] => 
                                                                        [level:protected] => 200
                                                                        [bubble:protected] => 1
                                                                        [formatter:protected] => Monolog\Formatter\LineFormatter Object
                                                                            (
                                                                                [format:protected] => [%datetime%] %channel%.%level_name%: %message% %context% %extra%

                                                                                [allowInlineLineBreaks:protected] => 1
                                                                                [ignoreEmptyContextAndExtra:protected] => 1
                                                                                [includeStacktraces:protected] => 
                                                                                [dateFormat:protected] => Y-m-d H:i:s
                                                                            )

                                                                        [processors:protected] => Array
                                                                            (
                                                                            )

                                                                    )

                                                            )

                                                        [processors:protected] => Array
                                                            (
                                                            )

                                                        [microsecondTimestamps:protected] => 1
                                                    )

                                            )

                                        [parameter] => Array
                                            (
                                                [$handler] => 
                                            )

                                    )

                                [1] => 
                            )

                        [1] => Array
                            (
                                [0] => Closure Object
                                    (
                                        [parameter] => Array
                                            (
                                                [$handler] => 
                                            )

                                    )

                                [1] => http_errors
                            )

                        [2] => Array
                            (
                                [0] => Closure Object
                                    (
                                        [parameter] => Array
                                            (
                                                [$handler] => 
                                            )

                                    )

                                [1] => allow_redirects
                            )

                        [3] => Array
                            (
                                [0] => Closure Object
                                    (
                                        [parameter] => Array
                                            (
                                                [$handler] => 
                                            )

                                    )

                                [1] => cookies
                            )

                        [4] => Array
                            (
                                [0] => Closure Object
                                    (
                                        [parameter] => Array
                                            (
                                                [$handler] => 
                                            )

                                    )

                                [1] => prepare_body
                            )

                    )

                [cached:GuzzleHttp\HandlerStack:private] => 
            )

        [allow_redirects] => Array
            (
                [max] => 5
                [protocols] => Array
                    (
                        [0] => http
                        [1] => https
                    )

                [strict] => 
                [referer] => 
                [track_redirects] => 
            )

        [http_errors] => 1
        [decode_content] => 1
        [verify] => 1
        [cookies] => 
        [headers] => Array
            (
                [User-Agent] => GuzzleHttp/6.2.1 curl/7.35.0 PHP/5.5.9-1ubuntu4.20
            )

    )

)

Thanks,

[fiboknacky]

Hello,

Could you please share your report definition or AWQL (and also if you included zero impressions or not)?
In addition, could you please clarif your client library version, PHP version and your environments?
Did this happen when you use ReportDownloader directly? Or do you have this issue when running the DownloadCriteriaReportWithAwql?

Best,
Knack

[chiragvels]

Hello Knack,

1. Report definition:
   Google\AdsApi\AdWords\Reporting\v201609\ReportDefinition Object
   (
   [selector:Google\AdsApi\AdWords\Reporting\v201609\ReportDefinition:private] => Google\AdsApi\AdWords\v201609\cm\Selector Object
   (
   [fields:protected] => Array
   (
   [0] => CampaignId
   [1] => Impressions
   [2] => Ctr
   [3] => AverageCpc
   [4] => AverageCpm
   [5] => Cost
   [6] => AveragePosition
   [7] => CostPerConversion
   [8] => ConversionRate
   [9] => Conversions
   [10] => AllConversions
   [11] => CostPerAllConversion
   [12] => CampaignName
   [13] => CampaignStatus
   [14] => Clicks
   [15] => AccountCurrencyCode
   [16] => ViewThroughConversions
   )
   [predicates:protected] => Array
   (
   [0] => Google\AdsApi\AdWords\v201609\cm\Predicate Object
   (
   [field:protected] => CampaignStatus
   [operator:protected] => IN
   [values:protected] => Array
   (
   [0] => ENABLED
   [1] => PAUSED
   )
   )
   )
   [dateRange:protected] =>
   [ordering:protected] =>
   [paging:protected] =>
   )
   [reportName:Google\AdsApi\AdWords\Reporting\v201609\ReportDefinition:private] => weekly_campaign_report #1488387248
   [reportType:Google\AdsApi\AdWords\Reporting\v201609\ReportDefinition:private] => CAMPAIGN_PERFORMANCE_REPORT
   [dateRangeType:Google\AdsApi\AdWords\Reporting\v201609\ReportDefinition:private] => LAST_WEEK
   [downloadFormat:Google\AdsApi\AdWords\Reporting\v201609\ReportDefinition:private] => CSV
   )

2. I am not including zero impressions.
   $reportSettings = (new ReportSettingsBuilder())
   ->fromFile($ads_ini_path)
   ->includeZeroImpressions(false)
   ->build();

3)client library version : v25.3.0, PHP version : PHP Version 5.5.9-1ubuntu4.20
environments : Linux sugar 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64

4. I am using the library and function as per example. Same thing working on my development environment on OSX, but giving issue on production environment on ubuntu.

please let me know if more info required.

Thanks,

[fiboknacky]

Hello,

I am using the library and function as per example. Same thing working on my development environment on OSX, but giving issue on production environment on ubuntu.

Do you mean the example I mentioned can run without issues, but the issues occurred when you tried to use the utility directly and happened only in Ubuntu?
And did this happen for all versions or only v201609?

And can you run other examples like GetCampaigns?
I asked because I'd like to confirm if the issue happens on using Guzzle only or not.

When you said you couldn't download the report, did the program just freeze?

Sorry for many questions.

Cheers,
Knack

[chiragvels]

Hello Kanck,

Do you mean the example I mentioned can run without issues, but the issues occurred when you tried to use the utility directly and happened only in Ubuntu?
->Yes, Examples under library run without issues, but not worked when implemented on my production environment.

And did this happen for all versions or only v201609?
-> I have checked in this version only.

And can you run other examples like GetCampaigns?
-> Yes I have checked some examples like account hierarchy and getCamapigns are worked fine.

I asked because I'd like to confirm if the issue happens on using Guzzle only or not.

When you said you couldn't download the report, did the program just freeze?
-> It just give me white page, no error, no exception, no response.

Thanks

[fiboknacky]

Hello @chiragvels

It looks like not the issue of ReportDownloader directly.
Perhaps there might be other issues underlying this.

Could you please send your code snippet and your client customer ID to adwordsapiadvisor@google.com please?
Include the message that links to this issue (#247) too please.

Thanks in advance.

Best,
Knack

[chiragvels]

Hello Knack,

Finally able to identify issue.

I am able to sorted out issue, under examples there is no logging enable but on my production I have enabled the logging and my log folder has permission 755, which is the issue, when I changed it to 777 and revert back to 755, it worked fine. Same with the report download folder. So may be issue with folder permission. I think this permission issue need to be managed by library?

[fiboknacky]

Hello @chiragvels,

The client library shouldn't modify or create anything on your file system without your intention, so I think this work as intended.
I'd recommend you to ensure the permission you specify as a location for logging is set appropriately.

Best,
Knack

[chiragvels]

Thanks Knack for reply.

What should be permission for log and reports download folder?

Thanks

[chiragvels]

Hello Knack,

Also noticed that I need to set reporting settings like includeZeroImpressions data or addHeader through api ini file or when preparing user sessions.

I feel that I cannot use the functionality better way If i need to download more reports/different reports with custom settings.

I mean that I have multiple clientCustomerId whose report I download on daily basis, in some case I need to pass includeZeroImpression(custom as per report) and in some case I need to pass some other settings.

For that I need to make session again for this only change.

In earlier API I can set that on reportDefination level. The same flexibility is missing for same session with different reporting settings.

Thanks,

[fiboknacky]

Hello @chiragvels

Thanks for your feedback.
First, the includeZeroImpressions option at report definition level itself is already sunset since v201603 (Please see Reporting in this release note), as it's duplicate with what we can do using HTTP headers (which is equivalent to ReportSettings in this library).

As for the usage of the library itself, do I understand correctly that if the clientCustomerId and reportSettings are settable in AdWordsSession, this can resolve your issue?

Best,
Knack

[chiragvels]

Hello Knack,

Yes, you are right includeZeroImpressions was removed from report definition, I mean to use like below, when we can set options to add ZeroImpressions or not.

    $options = array('version' => ADWORDS_VERSION);
$options['includeZeroImpressions'] = 'true';

Yes, I can set settings under AdWordsSession, like I did
$reportSettings = (new ReportSettingsBuilder())
->includeZeroImpressions(false)
->build();

        $session = (new AdWordsSessionBuilder())
                ->withOAuth2Credential($oAuth2Credential)
                ->withClientCustomerId($client_id)
                ->withReportSettings($reportSettings)
                ->build();

But my question is I need to set session again for same clientCustomerId for the report that required to include zero impression true or any other reporting related settings.

For for same clientCustomerId I have to create session 1) for includedZeroImpressions false like ->includeZeroImpressions(false)
and 2) for report which required includedZeroImpressions to true.
->includeZeroImpressions(true)

So For same customerId I have to make 2 session just for this settings. If we have options like earlier API than It would not required to have session again, I can just pass optional settings to $session.

Thanks,

[hakimio]

I agree that it would be nice to be able to change at least some AdWords session settings once it has already been built instead of having to completely rebuild it every time we need to change sth.

[chiragvels]

Hello Knack,

I have also noticed that when we downloading report lets say for last 15 days on daily basis.
Whenever there is no data (May be no impression at all on particular day) guzzle is not downloding report for that particular day randomly.

In my case I tried to download report for 10 clientCustomerId for last 15 days to test the new API. I failed to download at least 12 reports(In all above mentioned condition, No data), and guzzle not even gives error for that report.

Thanks,

[fiboknacky]

Hello @chiragvels

Could you please have a look at #246 too?
Probably it's the same issue and the workaround is to set timeout value larger like I described in the comment.

Best,
Knack

[chiragvels]

Ok Knack,

Let me try the workaround and will test again in some time and let you know.

In my case there is no large report (not even more than 1 MB), but let me try.

Thanks,

[chiragvels]

Hello Knack,

Not helped. Still missed the reports when downloading day by day report.

Seems like my issue was different than #246

As I said I am not able to download report with same session.

Like I have set the Adwords session once and try to download last 15 days data day by day. So total of 15 reports for one client customer ID.

Please let me know how I can achieve same.

Also tested with old library, I am not facing this issue with old library.

Thanks,

[fiboknacky]

Hello Chirag,

It's not common that there are no exceptions or errors at all.
Could you try setting error reporting to E_ALL?

And could you attach code snippet here? How do you loop to download reports?
I'd like to first reproduce your issue.

Cheers,
Knack

[chiragvels]

Hello Knack,

I have enabled the error reporting but still some of the reports are missing without giving exceptions or error.

Can you please provide me email, so that I can send you code snippet.

[fiboknacky]

Hello Chirag,

Please send to adwordsapiadvisor+thanet@google.com.
Thanks in advance!

Best,
Knack

[chiragvels]

Hello Knack,

I had tried this 4-5 times to produce exceptional error or PHP error, but not able to get the error, but same time not getting all the reports. In between missing some of the reports.

I am thinking that it may be issue while using custom date as start date and end date both as same date. But it is happening on randomly for client customerId .

Thanks,

[fiboknacky]

Hello Chirag,

Sorry for late reply.
I still don't have bandwidth to look at this in detail.
Will do as soon as possible.

Best,
Knack

[chiragvels]

Hello Knack,

Ok, I am still facing issue, checked latest today also.

Thanks,

[chiragvels]

Hello Knack,

Have you able to look into this?

Thanks

[fiboknacky]

Hi Chirag,

I'm sorry for late reply.
I have a look at your script but didn't find any misuse or bugs from using the ReportDownloader as well.

It looks like your missing reports are random, right?
There are no patterns of what kind of report definition or date range would fail this?
If this is true, then only possible factor that I can think of for now, is the connection issue between your computer and the API server.

Can you try making your cron script put some sleep time between each report download?
And just curious, how did you know that the script stops at the Guzzle HTTP request step if no errors were printed out at all?

Best,
Knack

[chiragvels]

Hello Knack,

It looks like your missing reports are random, right?
Yes, Missing Reports randomly.

There are no patterns of what kind of report definition or date range would fail this?
Yes, No patterns.

If this is true, then only possible factor that I can think of for now, is the connection issue between your computer and the API server.
No, It is not, because at same time I am also running the reports from OLD API, and its downloaded properly all the time.

Can you try making your cron script put some sleep time between each report download?
Let me try same, but it will increase time to finish job.

And just curious, how did you know that the script stops at the Guzzle HTTP request step if no errors were printed out at all?
-> I was receiving no errors, it is just skipping the report randomly. No exceptional error or nothing in the log as well.

Thanks,

[fiboknacky]

I was receiving no errors, it is just skipping the report randomly. No exceptional error or nothing in the log as well.

This is where I think we could get more clues.
When the script skips some reports, do you mean it just returned from the Guzzle's request() method immediately?
Is it possible for you to dig into it to see what's returned internally?

In many cases I see so far, if there's anything wrong about connection or timeout, it should produce some messages or exceptions.

If this is true, then only possible factor that I can think of for now, is the connection issue between your computer and the API server.
No, It is not, because at same time I am also running the reports from OLD API, and its downloaded properly all the time.

This library uses Guzzle and is thus quite different from that used in the old library.
So, when I say the connection issue, I meant connection between Guzzle's client and the API server connection, not just your computer. Sorry for miscommunication.

[chiragvels]

Hello Knack,

This is where I think we could get more clues.
When the script skips some reports, do you mean it just returned from the Guzzle's request() method immediately?
Is it possible for you to dig into it to see what's returned internally?
-> Yes, I have checked the Guzzle's request() method, and surprise to see that I was getting response here everytime under ReportDownloader.php, but I think the file is not returning to path provided. I am seeing that 18 objects were made, but against that only 16 files there under my files path.

$response = $this->httpClient->request(
'POST',
rtrim($this->session->getEndpoint(), '/')
. self::$REPORT_DOWNLOAD_URL_PATH,
$requestOptions
);

In many cases I see so far, if there's anything wrong about connection or timeout, it should produce some messages or exceptions.
-> It is always successful, so no error messages, but issue is files are not writing to folder passed while download report settings set.

[chiragvels]

Hello Knack,

Can you try making your cron script put some sleep time between each report download?
-> Yes, After adding Sleep of 1 seconds between each reports, I can able to pull all 40-50 reports without error, but this should be taken care by API?

I will try for more accounts tomorrow and let you know the final outcome.

Thanks,

[fiboknacky]

Hi Chirag,

This reminds me of the previous issue you've reported: file permission.
Have you checked that everything is right on your environments?
Can you also try changing the location to store reports?

Besides, you may want to investigate the saveToFile() method as well.

Best,
Knack

[chiragvels]

Hello Knack,
This reminds me of the previous issue you've reported: file permission.
Have you checked that everything is right on your environments?
-> I can able to download all the reports whenever I put sleep between reports, So not a file permission issue this time. Whenever tried to download reports in sequence like day by day, than missing the reports.
Can you also try changing the location to store reports?
-> I tried to check same, but same issue without using sleep.

Besides, you may want to investigate the saveToFile() method as well.
-> Ok, will try same and let you know the outcome.

[chiragvels]

Hello Knack,

Great help, Seems like adwords API is giving reports faster then anyone can imagine.
I am able to identify problem by looking at saveToFile(),
Problem with report name I am using time() to identify reports, but this is not enough to get reports.
So my reports are overlapping.

But still my report settings suggestions are to be consider. :)

[fiboknacky]

Yep, we'll consider that.

[fiboknacky]

Report settings override is supported in v28.0.0.