Java referer-parser doesn't work on Hadoop

[alexanderdean]

Unfortunately the bump to httpclient 4.3.3 has broken referer-parser on Hadoop.

Specifically it's this line of code:

81b88ff#diff-729b6a9a457c5e4a0b244bf130a1e08eR192

This is the error on Hadoop:

Caused by: java.lang.NoSuchMethodError: org.apache.http.client.utils.URLEncodedUtils.parse(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/util/List;
    at com.snowplowanalytics.refererparser.Parser.extractSearchTerm(Parser.java:205)
    at com.snowplowanalytics.refererparser.Parser.parse(Parser.java:154)
    at com.snowplowanalytics.refererparser.Parser.parse(Parser.java:116)
    at com.snowplowanalytics.refererparser.scala.Parser$.parse(Parser.scala:153)
    at com.snowplowanalytics.snowplow.enrich.common.enrichments.registry.RefererParserEnrichment.extractRefererDetails(RefererParserEnrichment.scala:107)
    at com.snowplowanalytics.snowplow.enrich.common.enrichments.EnrichmentManager$$anonfun$enrichEvent$2$$anonfun$apply$5.apply(EnrichmentManager.scala:291)
    at com.snowplowanalytics.snowplow.enrich.common.enrichments.EnrichmentManager$$anonfun$enrichEvent$2$$anonfun$apply$5.apply(EnrichmentManager.scala:277)
    at scala.Option.foreach(Option.scala:236)
    at com.snowplowanalytics.snowplow.enrich.common.enrichments.EnrichmentManager$$anonfun$enrichEvent$2.apply(EnrichmentManager.scala:277)
    at com.snowplowanalytics.snowplow.enrich.common.enrichments.EnrichmentManager$$anonfun$enrichEvent$2.apply(EnrichmentManager.scala:277)
    at scalaz.Validation$class.foreach(Validation.scala:126)
    at scalaz.Success.foreach(Validation.scala:329)
    at com.snowplowanalytics.snowplow.enrich.common.enrichments.EnrichmentManager$.enrichEvent(EnrichmentManager.scala:277)
    at com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$$anonfun$toCanonicalOutput$1$$anonfun$apply$2.apply(EtlJob.scala:70)
    at com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$$anonfun$toCanonicalOutput$1$$anonfun$apply$2.apply(EtlJob.scala:70)
    at scalaz.std.OptionFunctions$class.cata(Option.scala:157)
    at scalaz.std.option$.cata(Option.scala:209)
    at scalaz.syntax.std.OptionOps$class.cata(OptionOps.scala:9)
    at scalaz.syntax.std.ToOptionOps$$anon$1.cata(OptionOps.scala:103)
    at com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$$anonfun$toCanonicalOutput$1.apply(EtlJob.scala:70)
    at com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$$anonfun$toCanonicalOutput$1.apply(EtlJob.scala:70)
    at scalaz.Validation$class.flatMap(Validation.scala:141)
    at scalaz.Success.flatMap(Validation.scala:329)
    at com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$.toCanonicalOutput(EtlJob.scala:69)
    at com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$$anonfun$7.apply(EtlJob.scala:170)
    at com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$$anonfun$7.apply(EtlJob.scala:169)
    at com.twitter.scalding.MapFunction.operate(Operations.scala:58)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    ... 11 more

Hadoop bundles an old version of httpclient which doesn't have parse(String, Charset). There is talk of Hadoop removing that dependency but in any case we and EMR use an oldish version of Hadoop.

The problem is not using 4.3.3 per se, but using parse(String, Charset). /cc @squeed