Intelligently parse referrers so they're easier to glance at

[bamos]

Example:

• http://www.google.com/url?cmVhbGx5IGxvbmcgc3RyaW5nIHRvIGluZGljYXRlIGEgc2VjdXJlIHNlYXJjaA== toGoogle (secure search)`

[alexanderdean]

Can't you use the existing extracted referer fields from the RefererParserEnrichmentt?

[bamos]

Hi @alexanderdean - thanks!
I can use refrUrlhost and parse q=... out of refrUrlquery

Bing

pageReferrer = "http://www.bing.com/search?q=search+query&a=results&MID=2500", pageUrlscheme = "http", pageUrlhost = "bamos.github.io", pageUrlport = Just 80, pageUrlpath = "/XXXX", pageUrlquery = "", pageUrlfragment = "", refrUrlscheme = "http", refrUrlhost = "www.bing.com", refrUrlport = Just 80, refrUrlpath = "/search", refrUrlquery = "q=search+query&a=results&MID=2500", refrUrlfragment = "", refrMedium = "", refrSource = "", refrTerm = ""

Google

pageReferrer = "http://www.google.com/search?site=&source=hp&ei=...&q=search+query&oq=...&gs_l=...&q=search+query", pageUrlscheme = "http", pageUrlhost = "bamos.github.io", pageUrlport = Just 80, pageUrlpath = "/XXXX/", pageUrlquery = "", pageUrlfragment = "", refrUrlscheme = "http", refrUrlhost = "www.google.com", refrUrlport = Just 80, refrUrlpath = "/search", refrUrlquery = "site=&source=hp&ei=...&q=search+query&...", refrUrlfragment = "tbs=qdr:y&q=search+query", refrMedium = "", refrSource = "", refrTerm = ""

[alexanderdean]

scala> val refererUrl = "http://www.bing.com/search?q=search+query&a=results&MID=2500"
refererUrl: java.lang.String = http://www.bing.com/search?q=search+query&a=results&MID=2500

scala> val pageUrl    = "http://random"
pageUrl: java.lang.String = http://random

scala> import com.snowplowanalytics.refererparser.scala.Parser
import com.snowplowanalytics.refererparser.scala.Parser

scala> for (r <- Parser.parse(refererUrl, pageUrl)) {
     |   println(r.medium)
     |   for (s <- r.source) {
     |     println(s)
     |   }
     |   for (t <- r.term) {
     |     println(t)
     |   }
     | }
search
Bing
search query

Therefore I think your RefererParserEnrichment inside your Kinesis Enrich isn't correctly configured.

[bamos]

I see - thanks! Great info.

[bamos]

@alexanderdean - the latest snowplow's referrer enrichments are also working great.
Leaving this issue open so I can perfect my prettyReferrer function as
I look at new events with the new enrichments.

prettyReferrer:: EnrichedEvent -> Maybe String
prettyReferrer e =
    case (pageReferrer e, refrMedium e, refrSource e, refrTerm e) of
      ("", _, _, _) -> Nothing
      (url, "search", "", term) -> Just $ url ++ " [TODO: prettyReferrer: \
                                                 \empty refrSource?]"
      (url, "search", source, "") -> Just $ source ++ " [Secure Search]"
      (url, "search", source, term) -> Just $ concat [source, " Search - ", term]
      (url, med, source, term) -> Just $ "TODO: prettyReferrer: " ++
                                    intercalate " - " [url, med, source, term]

[alexanderdean]

Ah, great to hear it!