Problem parsing google abuse reports #235

CrazyLlama · 2017-07-20T09:41:41Z

https://github.com/AbuseIO/parser-google

Hi everyone,

Just wondering if anyone else has the same problem as I do, constant failing google abuse reports (SBR) failing to be parsed.

I'm doing some troubleshooting now but I think there may be a few possible source:

DKIM parsing problems which causes the parsing to fail (In which case you may be able to make use of something such as https://github.com/angrychimp/php-dkim which can parse the DKIM or build something to do so if this is the problem )
Multiple "Received:" Headers?

If anyone's experienced this previously or has any ideas I'd love some input!

Dank je 😄

kruisdraad · 2017-07-20T09:44:01Z

perhaps some kind of information like error logs might help.

for me it works fine

CrazyLlama · 2017-07-20T09:55:29Z

Error seems to be as follows in framework.log:

Jul 20 10:46:18 localhost abuseio[4768]: production.ERROR: exception 'ErrorException' with message 'simplexml_load_string(): Entity: line 1: parser error : Start tag expected, '<' not found' in /op
t/abuseio/vendor/abuseio/parser-google/src/Google.php:31 Stack trace: #0 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'simplexml_load_...', '/opt/abuseio/ve
...', 31, Array) #1 /opt/abuseio/vendor/abuseio/parser-google/src/Google.php(31): simplexml_load_string('Google detected...') #2 /opt/abuseio/app/Jobs/EmailProcess.php(153): AbuseIO\Parsers\Google-
>parse() #3 [internal function]: AbuseIO\Jobs\EmailProcess->handle() #4 /opt/abuseio/bootstrap/cache/compiled.php(1187): call_user_func_array(Array, Array) #5 /opt/abuseio/bootstrap/cache/compiled.
php(9505): Illuminate\Container\Container->call(Array) #6 [internal function]: Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(AbuseIO\Jobs\EmailProcess)) #7 /opt/abuseio/bootstrap/cache
/compiled.php(9657): call_user_func(Object(Closure), Object(AbuseIO\Jobs\EmailProcess)) #8 [internal function]: Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(AbuseIO\Jobs\Email
Process)) #9 /opt/abuseio/bootstrap/cache/compiled.php(9639): call_user_func(Object(Closure), Object(AbuseIO\Jobs\EmailProcess)) #10 /opt/abuseio/bootstrap/cache/compiled.php(9512): Illuminate\Pipe
line\Pipeline->then(Object(Closure)) #11 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(43): Illuminate\Bus\Dispatcher->dispatchNow(Object(AbuseIO\Jobs\EmailProces
s), Object(Closure)) #12 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(129): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\DatabaseJob), Array) #13
 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Jobs/DatabaseJob.php(50): Illuminate\Queue\Jobs\Job->resolveAndFire(Array) #14 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue
/Worker.php(218): Illuminate\Queue\Jobs\DatabaseJob->fire() #15 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(160): Illuminate\Queue\Worker->process('database', Object(Illum
inate\Queue\Jobs\DatabaseJob), '1', '0') #16 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(111): Illuminate\Queue\Worker->pop(NULL, 'abuseio_email_i...', '0', '3', '1') #17
/opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(87): Illuminate\Queue\Worker->runNextJobForDaemon(NULL, 'abuseio_email_i...', '0', '3', '1') #18 /opt/abuseio/vendor/laravel/fr
amework/src/Illuminate/Queue/Console/WorkCommand.php(103): Illuminate\Queue\Worker->daemon(NULL, 'abuseio_email_i...', '0', '256', '3', '1') #19 /opt/abuseio/vendor/laravel/framework/src/Illuminate
/Queue/Console/WorkCommand.php(71): Illuminate\Queue\Console\WorkCommand->runWorker(NULL, 'abuseio_email_i...', '0', '256', true) #20 [internal function]: Illuminate\Queue\Console\WorkCommand->fire
() #21 /opt/abuseio/bootstrap/cache/compiled.php(1187): call_user_func_array(Array, Array) #22 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Console/Command.php(150): Illuminate\Container\Co
ntainer->call(Array) #23 /opt/abuseio/vendor/symfony/console/Command/Command.php(263): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Componen
t\Console\Output\ConsoleOutput)) #24 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\I
nput\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #25 /opt/abuseio/vendor/symfony/console/Application.php(852): Illuminate\Console\Command->run(Object(Symfony\Component\Conso
le\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #26 /opt/abuseio/vendor/symfony/console/Application.php(199): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #27 /opt/abuseio/vendor/symfony/console/Application.php(123): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #28 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(107): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #29 /opt/abuseio/artisan(36): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #30 {main}

Which appears to refer to the following in Google.php:

$xml = simplexml_load_string($this->parsedMail->getMessageBody());

Which is strange?

Park0 · 2017-07-20T12:53:16Z

Made it a bit more readable

Jul 20 10:46:18 localhost abuseio[4768]: production.
ERROR: exception 'ErrorException' with message 'simplexml_load_string(): 
Entity: line 1: parser error : Start tag expected, '<' not found' in /opt/abuseio/vendor/abuseio/parser-google/src/Google.php:31 
  Stack trace: 
    #0 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'simplexml_load_...', '/opt/abuseio/ve ...', 31, Array) 
	#1 /opt/abuseio/vendor/abuseio/parser-google/src/Google.php(31): simplexml_load_string('Google detected...') 
	#2 /opt/abuseio/app/Jobs/EmailProcess.php(153): AbuseIO\Parsers\Google->parse() 
	#3 [internal function]: AbuseIO\Jobs\EmailProcess->handle() 
	#4 /opt/abuseio/bootstrap/cache/compiled.php(1187): call_user_func_array(Array, Array) 
	#5 /opt/abuseio/bootstrap/cache/compiled.php(9505): Illuminate\Container\Container->call(Array) 
	#6 [internal function]: Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(AbuseIO\Jobs\EmailProcess)) 
	#7 /opt/abuseio/bootstrap/cache/compiled.php(9657): call_user_func(Object(Closure), Object(AbuseIO\Jobs\EmailProcess)) 
	#8 [internal function]: Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(AbuseIO\Jobs\EmailProcess)) 
	#9 /opt/abuseio/bootstrap/cache/compiled.php(9639): call_user_func(Object(Closure), Object(AbuseIO\Jobs\EmailProcess)) 
	#10 /opt/abuseio/bootstrap/cache/compiled.php(9512): Illuminate\Pipeline\Pipeline->then(Object(Closure)) 
	#11 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(43): Illuminate\Bus\Dispatcher->dispatchNow(Object(AbuseIO\Jobs\EmailProcess), Object(Closure)) 
	#12 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(129): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\DatabaseJob), Array) 
	#13 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Jobs/DatabaseJob.php(50): Illuminate\Queue\Jobs\Job->resolveAndFire(Array) 
	#14 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(218): Illuminate\Queue\Jobs\DatabaseJob->fire() 
	#15 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(160): Illuminate\Queue\Worker->process('database', Object(Illuminate\Queue\Jobs\DatabaseJob), '1', '0')
	#16 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(111): Illuminate\Queue\Worker->pop(NULL, 'abuseio_email_i...', '0', '3', '1')
	#17 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(87): Illuminate\Queue\Worker->runNextJobForDaemon(NULL, 'abuseio_email_i...', '0', '3', '1') 
	#18 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(103): Illuminate\Queue\Worker->daemon(NULL, 'abuseio_email_i...', '0', '256', '3', '1')
	#19 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(71): Illuminate\Queue\Console\WorkCommand->runWorker(NULL, 'abuseio_email_i...', '0', '256', true) 
	#20 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()
	#21 /opt/abuseio/bootstrap/cache/compiled.php(1187): call_user_func_array(Array, Array)
	#22 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Console/Command.php(150): Illuminate\Container\Container->call(Array)
	#23 /opt/abuseio/vendor/symfony/console/Command/Command.php(263): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
	#24 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
	#25 /opt/abuseio/vendor/symfony/console/Application.php(852): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
	#26 /opt/abuseio/vendor/symfony/console/Application.php(199): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
	#27 /opt/abuseio/vendor/symfony/console/Application.php(123): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
	#28 /opt/abuseio/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(107): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
	#29 /opt/abuseio/artisan(36): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
	#30 {main}

kruisdraad · 2017-07-21T08:05:18Z

I am setting this to a duplicate of #217

Its not a problem in the parser, but within the handler's core.

Can you confirm you are using a non english type system or email body?

CrazyLlama · 2017-07-21T08:45:21Z

All the failed emails are English on UK English systems, I'll get an example of the failed emails up with all the IPs/URLs changed (unless they're non-english characters):

CrazyLlama · 2017-07-21T08:55:00Z

Return-Path: <abuse@isp.co.uk>
Received: from email.isp.com (cashub1.isp.com [xxx.xxx.xxx.xxx])
        by localhost (Postfix) with ESMTPS id 5A04240284
        for <notifier@abuse.isp.co.uk>; Wed, 19 Jul 2017 07:21:46 +0100 (BST)
Resent-From: <abuse@isp.co.uk>
Received: from filter.mail.co.uk (xxx.xxx.xxx.xxx) by
 CASHUB1.office.isp (xxx.xxx.xxx.xxx) with Microsoft SMTP Server (TLS) id
 14.3.352.0; Wed, 19 Jul 2017 07:21:46 +0100
Received: from mail-pg0-f71.google.com ([74.125.83.71]) by
 filter2.magicmail.co.uk with esmtps (TLSv1:AES128-SHA:128)     (Exim 4.77)
        (envelope-from
 <3Z_puWQcKBegXYbOZViQYYQVO.MYWKLecOeUPKcd.MY.eU@malware.bounces.google.com>)
        id 1dXiMl-0008Ec-4j     for abuse@isp.co.uk; Wed, 19 Jul 2017 07:21:48 +0100
Received: by mail-pg0-f71.google.com with SMTP id 125so45071140pgi.2
        for <abuse@isp.co.uk>; Tue, 18 Jul 2017 23:21:43 -0700 (PDT)
X-Gm-Message-State: AIVw111M+u60UaFDH1Bqd68fxnTNI7upY8+zpYZGtfJ/YGk70hBReBVl
        L3uNpm+BajXF75DWyOKdVQfnTsQMyjNU63xtQBGgQwgfZkj1CyxiQCM0yCdxqO2JC7MKYKFZg/9
        eDtWzBiA=
MIME-Version: 1.0
X-Received: by xxx.xxx.xxx.xxx with SMTP id u4mr676162pfk.10.1500445287554; Tue,
 18 Jul 2017 23:21:27 -0700 (PDT)
Date: Tue, 18 Jul 2017 23:21:27 -0700
Message-ID: <a101234b67212345e0554a5a2@google.com>
Subject: Safe Browsing Report for ASNxxxx (isp) on 07/18/2017
From: Google Safe Browsing <noreply@google.com>
To: <abuse@isp.co.uk>
Content-Type: text/plain; charset="UTF-8"; format=flowed; delsp=yes
X-Spam-Reject: 50
X-Spam-Tag: 30
X-Spam-Score: -1.4
X-Spam-Score-Int: -13
X-Spam-Report: Tests run=BAYES_00,RCVD_IN_DNSWL_NONE,RCVD_IN_SORBS_SPAM,RP_MATCHES_RCVD,SPF_PASS

Google detected 5 suspicious URLs (space inserted to prevent accidental
clicking in case your email client auto-links URLs):

http://sub-domain.sub-domain.url .co.uk/dir/ (xxx.xxx.xxx.xxx)
http://sub-domain.sub-domain.url .co.uk/ (xxx.xxx.xxx.xxx)
http://sub-domain.sub-domain.url .co.uk/file.html?REDACTED
(xxx.xxx.xxx.xxx)
http://www.url .co.uk/dir/ (xxx.xxx.xxx.xxx)
http://www.url .co.uk/dir/file
(xxx.xxx.xxx.xxx)


Google detected 4 social engineering URLs (space inserted to prevent
accidental clicking in case your email client auto-links URLs):

http://sub-domain.sub-domain.url .com/dir/file.cfm?REDACTED (xxx.xxx.xxx.xxx)
http://sub-domain.url .co.uk/cov/office/ (xxx.xxx.xxx.xxx)
http://www.url .co.uk/chase/dir/file.php?REDACTED
(xxx.xxx.xxx.xxx)
https://sub-domain.url .co.uk/dir/dir/ (xxx.xxx.xxx.xxx)

CrazyLlama · 2017-07-21T08:57:39Z

I checked for non-ascii characters using vim as well

:set hlsearch

/[^\x00-\x7F]

Pattern not found: [^\x00-\x7F]

kruisdraad · 2017-07-21T09:07:10Z

right, you have the human readable version of the google report. You must use the XML type reporting, that will work a lot better ;)

kruisdraad · 2017-07-21T09:08:25Z

Have a look at the sample dir, it hold the google sample too:

https://github.com/AbuseIO/AbuseIO/blob/master/extra/notifier-samples/google/Safe_Browsing_Report_1.eml

Thats the format you need, its a XML setting the de SBR in the portal

CrazyLlama · 2017-07-21T09:22:10Z

Aaaaaah, I see, it's blindingly obvious now hahaha.

Thank you so much for your help Bart!

kruisdraad · 2017-07-21T12:18:56Z

It should not stack trace like that we nee to groacefully exit it

kruisdraad · 2017-10-07T18:45:28Z

validation added in version 1.3.8

kruisdraad marked this as a duplicate of #217 Jul 21, 2017

CrazyLlama closed this as completed Jul 21, 2017

kruisdraad reopened this Jul 21, 2017

kruisdraad closed this as completed Oct 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem parsing google abuse reports #235

Problem parsing google abuse reports #235

CrazyLlama commented Jul 20, 2017 •

edited

kruisdraad commented Jul 20, 2017

CrazyLlama commented Jul 20, 2017 •

edited

Park0 commented Jul 20, 2017

kruisdraad commented Jul 21, 2017

CrazyLlama commented Jul 21, 2017 •

edited

CrazyLlama commented Jul 21, 2017

CrazyLlama commented Jul 21, 2017 •

edited

kruisdraad commented Jul 21, 2017

kruisdraad commented Jul 21, 2017

CrazyLlama commented Jul 21, 2017 •

edited

kruisdraad commented Jul 21, 2017

kruisdraad commented Oct 7, 2017

Problem parsing google abuse reports #235

Problem parsing google abuse reports #235

Comments

CrazyLlama commented Jul 20, 2017 • edited

kruisdraad commented Jul 20, 2017

CrazyLlama commented Jul 20, 2017 • edited

Park0 commented Jul 20, 2017

kruisdraad commented Jul 21, 2017

CrazyLlama commented Jul 21, 2017 • edited

CrazyLlama commented Jul 21, 2017

CrazyLlama commented Jul 21, 2017 • edited

kruisdraad commented Jul 21, 2017

kruisdraad commented Jul 21, 2017

CrazyLlama commented Jul 21, 2017 • edited

kruisdraad commented Jul 21, 2017

kruisdraad commented Oct 7, 2017

CrazyLlama commented Jul 20, 2017 •

edited

CrazyLlama commented Jul 20, 2017 •

edited

CrazyLlama commented Jul 21, 2017 •

edited

CrazyLlama commented Jul 21, 2017 •

edited

CrazyLlama commented Jul 21, 2017 •

edited