Bad rows, regex fails to parse user-agent.

[pkallos]

I am seeing about 5% of my records marked as bad because the user agent parsing fails. Example:

{
    "line": {
        "timestamp": 1395299873568,
        "collector": "ssc-0.1.0-kinesis",
        ...
        "headers": [..., "User-Agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko", ...],
        "networkUserId": "7e98e0a0-e316-40ae-94f4-5a723264397c",
        ....
        "setNetworkUserId": true
    },
    "errors": ["Exception parsing useragent [Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko]: [No group 1]"]
}

[alexanderdean]

Super-weird, scheduling... @yalisassoon have you ever seen this before?

[alexanderdean]

Two things I can think:

1. We bumped the version of our useragent library recently
2. Maybe the Scala Stream Collector is having some kind of issue recording user agents

[yalisassoon]

I can't ever remember seeing that error...
I have double checked and can't find it in any of our records for either
Snowplow or Psychic Bazaar. Does make me wonder if it's a Scala Stream
Collector specific issue (maybe to do with the encoding of the user agent
string?)

On Thu, Mar 20, 2014 at 9:17 AM, Alexander Dean notifications@github.comwrote:

Two things I can think:

We bumped the version of our useragent library recently
Maybe the Scala Stream Collector is having some kind of issue recording user agents

Reply to this email directly or view it on GitHubhttps://github.com//issues/576#issuecomment-38147190
.

Co-founder
Snowplow Analytics http://snowplowanalytics.com/
The Roma Building, 32-38 Scrutton Street, London EC2A 4RQ, United Kingdom
+44 (0)203 589 6116
+44 7841 954 117
@yalisassoon https://twitter.com/yalisassoonhttps://twitter.com/yalisassoon

[alexanderdean]

Agree

[pkallos]

OK so it turns out this was browser useragent utils failing to recognize IE 11's useragent.

I first took a stab at #62 but the deltas between ua_parser and the existing library are pretty significant.

ua_parser doesn't report as many fields so the clientattributes would have to be paired down to

   case class ClientAttributes(
       // Browser
-      browserName: String,
       browserFamily: String,
-      browserVersion: Option[String],
-      browserType: String,
-      browserRenderEngine: String,
+      browserVersion: String,
       // OS the browser is running on
-      osName: String,
       osFamily: String,
-      osManufacturer: String,
       // Hardware the OS is running on
       deviceType: String,
       deviceIsMobile: Boolean)

not sure if worth doing but in my case losing the convenience of "browserType" -> Computer, Tablet, Mobile, ... is kind of a non-starter.

Will open a PR that addresses this ticket.

[alexanderdean]

Thanks Phil - we expected there would be a mismatch between the two UA parsers - bit of a shame though that ua-parser loses the convenient browserType field. Will add a comment to #62

[fblundun]

Can I close this? I think it's covered by #62 and #792.

[alexanderdean]

Plus the underlying bug was fixed in user-agent utils 1.12, which @pkallos pulled in via #662. So yes let's close.

[alexanderdean]

Cleared milestone too