Skip to content
Library for extracting marketing attribution data from referrer URLs http://snowplowanalytics.com
C# Scala Java Ruby Python JavaScript Other
Find file
Pull request Compare This branch is 55 commits ahead, 166 commits behind snowplow:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
dotnet
java-scala
nodejs
python
ruby
.travis.yml
README.md
referers.yml
sync_yaml.sh

README.md

referer-parser

Java/Scala: Build Status

referer-parser is a multi-language library for extracting marketing attribution data (such as search terms) from referer URLs, inspired by the ua-parser project (an equivalent library for user agent parsing).

referer-parser is a core component of Snowplow, the open-source web-scale analytics platform powered by Hadoop, Hive and Redshift.

Note that we always use the original HTTP misspelling of 'referer' (and thus 'referal') in this project - never 'referrer'.

Maintainers

Usage: Java

The Java version of this library uses the updated API, and identifies search, social, webmail, internal and unknown referers:

import com.snowplowanalytics.refererparser.Parser;

...

String refererUrl = "http://www.google.com/search?q=gateway+oracle+cards+denise+linn&hl=en&client=safari";
String pageUrl    = "http:/www.psychicbazaar.com/shop"; // Our current URL

Parser refererParser = new Parser();
Referer r = refererParser.parse(refererUrl, pageUrl);

System.out.println(r.medium);     // => "search"
System.out.println(r.source);     // => "Google"
System.out.println(r.term);       // => "gateway oracle cards denise linn"

For more information, please see the Java/Scala README.

Usage: Scala

The Scala version of this library uses the updated API, and identifies search, social, webmail, internal and unknown referers:

val refererUrl = "http://www.google.com/search?q=gateway+oracle+cards+denise+linn&hl=en&client=safari"
val pageUrl    = "http:/www.psychicbazaar.com/shop" // Our current URL

import com.snowplowanalytics.refererparser.scala.Parser
for (r <- Parser.parse(refererUrl, pageUrl)) {
  println(r.medium)         // => "search"
  for (s <- r.source) {
    println(s)              // => "Google"
  }
  for (t <- r.term) {
    println(t)              // => "gateway oracle cards denise linn"
  }
}

For more information, please see the Java/Scala README.

Usage: Ruby

The Ruby version of this library still uses the old API, and identifies search referers only:

require 'referer-parser'

referer_url = 'http://www.google.com/search?q=gateway+oracle+cards+denise+linn&hl=en&client=safari'

r = RefererParser::Referer.new(referer_url)

puts r.known?                # =>  true
puts r.referer               # => 'Google'
puts r.search_parameter      # => 'q'     
puts r.search_term           # => 'gateway oracle cards denise linn'
puts r.uri.host              # => 'www.google.com'

For more information, please see the Ruby README.

Usage: Python

Create a new instance of a Referer object by passing in the url you want to parse:

from referer_parser import Referer

referer_url = 'http://www.google.com/search?q=gateway+oracle+cards+denise+linn&hl=en&client=safari'

r = Referer(referer_url)

The r variable now holds a Referer instance. The important attributes are:

print(r.known)              # True
print(r.referer)            # 'Google'
print(r.medium)             # 'search'
print(r.search_parameter)   # 'q'     
print(r.search_term)        # 'gateway oracle cards denise linn'
print(r.uri)                # ParseResult(scheme='http', netloc='www.google.com', path='/search', params='', query='q=gateway+oracle+cards+denise+linn&hl=en&client=safari', fragment='')

Optionally, pass in the current URL as well, to handle internal referers

from referer_parser import Referer

referer_url = 'http://www.snowplowanalytics.com/about/team'
curr_url = 'http://www.snowplowanalytics.com/account/profile'

r = Referer(referer_url, curr_url)

For more information, please see the Python README.

Usage: node.js

The node.js (JavaScript) version of this library uses a hybrid of the new and old API, and identifies search, social, webmail, internal and unknown referers:

Create a new instance of a Referer object by passing in the url you want to parse:

var Referer = require('referer-parser')

referer_url = 'http://www.google.com/search?q=gateway+oracle+cards+denise+linn&hl=en&client=safari'

var r = new Referer(referer_url)

The r variable now holds a Referer instance.

Optionally, pass in the current URL as well, to handle internal referers

var Referer = require('referer-parser')

var referer_url = 'http://www.snowplowanalytics.com/about/team'
var current_url = 'http://www.snowplowanalytics.com/account/profile'

var r = Referer(referer_url, current_url)

For more information, please see the node.js README.

Usage: .NET

The .NET (C#) version of this library uses the updated API, and identifies search, social, webmail, internal and unknown referers:

using RefererParser;

...

string refererUrl = "http://www.google.com/search?q=gateway+oracle+cards+denise+linn&hl=en&client=safari";
string pageUrl    = "http:/www.psychicbazaar.com/shop"; // Our current URL

var referer = Parser.Parse(new Uri(refererUrl), pageUrl);

Console.WriteLine(r.Medium); // => "Search"
Console.WriteLine(r.Source); // => "Google"
Console.WriteLine(r.Term); // => "gateway oracle cards denise linn"

For more information, please see the .NET README.

referers.yml

referer-parser identifies whether a URL is a known referer or not by checking it against the referers.yml file; the intention is that this YAML file is reusable as-is by every language-specific implementation of referer-parser.

The file is broken out into sections for the different mediums that we support:

  • unknown for when we know the source, but not the medium
  • email for webmail providers
  • social for social media services
  • search for search engines

Then within each section, we list each known provider (aka source) by name, and then which domains each provider uses. For search engines, we also list the parameters used in the search engine URL to identify the search term. For example:

Google: # Name of search engine referer
  parameters:
    - 'q' # First parameter used by Google
    - 'p' # Alternative parameter used by Google
  domains:
    - google.co.uk  # One domain used by Google
    - google.com    # Another domain used by Google
    - ...

The number of referers and the domains they use is constantly growing - we need to keep referers.yml up-to-date, and hope that the community will help!

Contributing

We welcome contributions to referer-parser:

  1. New search engines and other referers - if you notice a search engine, social network or other site missing from referers.yml, please fork the repo, add the missing entry and submit a pull request
  2. Ports of referer-parser to other languages - we welcome ports of referer-parser to new programming languages (e.g. PHP, Go, Haskell, C)
  3. Bug fixes, feature requests etc - much appreciated!

Please sign the Snowplow CLA before making pull requests.

Support

General support for referer-parser is handled by the team at Snowplow Analytics Ltd.

You can contact the Snowplow Analytics team through any of the channels listed on their wiki.

Copyright and license

referers.yml is based on Piwik's SearchEngines.php and Socials.php, copyright 2012 Matthieu Aubry and available under the GNU General Public License v3.

The original Ruby code is copyright 2012-2013 Snowplow Analytics Ltd and is available under the Apache License, Version 2.0.

The Java/Scala port is copyright 2012-2013 Snowplow Analytics Ltd and is available under the Apache License, Version 2.0.

The Python port is copyright 2012-2013 Don Spaulding and is available under the Apache License, Version 2.0.

The node.js (JavaScript) port is copyright 2013 Martin Katrenik and is available under the Apache License, Version 2.0.

The .NET (C#) port is copyright 2013 iPerform Software and is available under the Apache License, Version 2.0.

Something went wrong with that request. Please try again.