Generates string tokens as alternative IDs for easy reference.
Clone or download
bkuhlmann Updated project changes to use semantic versions.
Keeps project changes/history labeled with consistent version labels.

[ci skip]
Latest commit ef006a1 Apr 1, 2018
Failed to load latest commit information.
.github Updated GitHub templates. Jun 29, 2017
lib Added version release changes. Apr 1, 2018
spec Refactored temp dir shared context as a pathname. Mar 25, 2018
.codeclimate.yml Updated Code Climate configuration to Version 2.0.0. Dec 10, 2017
.gitignore Removed `.bundle` directory from `.gitignore`. Feb 4, 2017
.rubocop.yml Updated Rubocop configuration. Apr 29, 2017
.todo.reek Updated Reek TODO list to ignore IrresponsibleModule. Dec 5, 2017 Updated to Code of Conduct, Version 1.4.0. Feb 13, 2016 Updated CONTRIBUTING documentation. Jun 25, 2017
Gemfile Added frozen string literal pragma. Jul 13, 2016
Guardfile Updated to Apache 2.0 license. Dec 31, 2017 Removed Gemnasium support. Feb 4, 2018
Rakefile Added Bundler Audit gem. Sep 23, 2017
circle.yml Removed Circle CI Bundler cache. Mar 31, 2018
tokener.gemspec Updated to Gemsmith 12.0.0. Apr 1, 2018


Gem Version Code Climate Maintainability Code Climate Test Coverage Circle CI Status

Generates string tokens as alternative IDs for easy reference. This can prove useful for URL addresses, internal code lookup, human readability, file system files, etc.

Table of Contents


  • Supports ActiveRecord models.
  • Supports token generation from single/multiple attributes (including delimiter customization).
  • Supports custom token attribute naming. By default, the token attribute is used.
  • Supports token validation using the similar criteria to token generation.
  • Supports token scopes.
  • Supports token exclude lists.
  • Supports custom token transforms. By default, tokens are transliterated, URI encoded, and URI tokenized. Additionally, tokens can be decoded and detokenized.
  • Supports custom unique token generation. By default, duplicate tokens are suffixed with a 32-character hex. Example: "-3885821dc01e3df51b10e08df5c007d1".
  • Supports finding records by token, primary key, or custom key.
  • Supports rebuilding of tokens either gracefully or forcefully.


  1. Ruby 2.5.x
  2. Ruby on Rails 5.x.x


Type the following to install:

gem install tokener

Add the following to your Gemfile:

gem "tokener"


The are many options available to you, see below.

Single Attribute

Basic, out-of-the-box usage would be as follows:

class Song < ActiveRecord::Base
  tokener :label

The resulting attribute=value pairs would yield: {label: "Can't You Hear Me Knocking", token: "cant_you_hear_me_knocking"}.

Multiple Attributes

When a single attribute is not enough, use multiple attributes. Example (first_name = "Jimi" and last_name = "Hendrix"):

class Musician < ActiveRecord::Base
  tokener [:first_name, :last_name]

The resulting token would be: "jimi_hendrix".

Attribute order is important as the reverse is possible too:

class Musician < ActiveRecord::Base
  tokener [:last_name, :first_name]

The resulting token would be:: "hendrix_jimi".

Multiple Attribute Custom Delimiter

When the default delimiter does not suffix, it can be customized:

class Musician < ActiveRecord::Base
  tokener [:first_name, :last_name], delimiter: '-'

The resulting token would be: "jimi-hendrix".

Custom Token

If the default behavior is not to your liking then the ability to define a specific token attribute is possible. Example:

class Song < ActiveRecord::Base
  tokener :label, token: :slug

The resulting attribute/value pairs would be: {label: "Can't You Hear Me Knocking", slug: "cant_you_hear_me_knocking"}.


Token validation is performed prior to saving, using similar criteria used in creation. By default, this results in the following (example):

validates :token,
          presence: true,
          format: {
            with: /\A[0-9a-zA-Z\-\_\%]+\z/,
            message: "must be digits, lowercase characters, dashes, and/or underscores."

If scope is provided, then the following would be added (example):

uniqueness: {scope: [:category_id]}

If an exclude list is provided, then the following would be added (example):

exclusion: ["one"]

NOTE: The capital letters and the percent (%) are allowed in validation to support URI encoded characters.

TIP: These are the validation settings, if enabled, by this gem. Should these settings not cover specific requirements, then the option of applying custom validations to your class can be used in addition to those provide by this gem.


Some situations might require the exclusion of reserved tokens. This can be done by defining an exclude list as follows:

class Song < ActiveRecord::Base
  tokener :label, excludes: %w(rocky_raccoon helter_skelter revolution_9)

Any generated token matching a excluded token would result in a validation error.


Scopes allow token generation to be limited to single/multiple attributes. Example:

class Album < ActiveRecord::Base
  tokener :title, scope: :genre_id

This would limit the token to be unique for with the same genre. Based on the definition above, the following would be valid:

  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 1
  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 2

However, it would be invalid if the genre ID was 1 for both records.

More than one scope is also available by passing in an array to the scope. Example:

class Album < ActiveRecord::Base
  tokener :title, scope: [:genre_id, :category_id]

Based on the example above, the following would be valid:

  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 1, category_id: 1
  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 2, category_id: 1

Using more than one of the same genre_id and category_id combinations would throw a validation error.


Transformation of raw input into a token is performed via the Tokener::Transformer::Default class. This class provides the default implementation for defining what unicode characters are allowed for tokenization, transliteration, and encoding. It also provides methods for detokenization and decoding. To customize, extend the Tokener::Transformer::Base abstract class and use the Tokener::Transformer::Default implementation as an example. Once a custom implementation has been built, it can be defined via the tokener macro. Example:

class Example < ActiveRecord::Base
  tokener :label, transformer:


By default, the ability to auto-generate unique tokens is disabled. When enabled, unique tokens are always generated after the first occurance. Example:

class Song < ActiveRecord::Base
  tokener :label, token: :slug, unique: true

The following tokens would be generated (with a unique, 32-character, hex applied after the first occurance). Example:

  • 1st Occurance: "cant_you_hear_me_knocking"
  • 2nd Occurance: "cant_you_hear_me_knocking-3885821dc01e3df51b10e08df5c007d1"
  • 3rd Occurance: "cant_you_hear_me_knocking-d2065aab58203c56440e8693e669933e"

Generation of unique suffixes can be customized by extending the Tokener::Uniquer::Base abstract class. Use the Tokener::Uniquer::Hex implementation as an example. When ready to use a custom uniquer, define it via the tokener macro as follows:

class Example < ActiveRecord::Base
  tokener :label, uniquer: MyCustomUniquer


The ability to search by token or ID is supported via the .tokener_find method. The following will yield the same result:

Song.tokener_find "cant_you_hear_me_knocking"
Song.tokener_find 1

The token is searched first, otherwise defaulting to the original ID (integer) or UUID (string) if enabled.

It is recommended that the .tokener_find method be used when finding records as it makes finding encoded tokens easy. Example:

# Finds the URI encoded version (i.e. "%E4%BD%A0%E8%83%BD%E5%90%AC%E5%88%B0%E6%88%91%E7%9A%84%E6%95%B2%E9%97%A8") stored in the database.
Song.tokener_find "你能听到我的敲门"

# Accepts raw encoding too.
Song.tokener_find "%E4%BD%A0%E8%83%BD%E5%90%AC%E5%88%B0%E6%88%91%E7%9A%84%E6%95%B2%E9%97%A8"

When using .tokener_find and a record is not found, it will answer nil. If this is not desired and you'd like to raise an ActiveRecord::RecordNotFound error, use tokener_find! instead.


Parameterization of a tokenized record is provided via the #to_param method (common to all ActiveRecord objects but enhanced by this gem). If a token value exists, the token is returned (otherwise the original ID is returned). Example:

# When token exists.

# When token exists and is encoded (is decoded before answering).

# When no token exists and primary ID is an integer.

# When no token exists and primary ID is an UUID.

These makes generating pretty URLs easy. Example:


If the default #to_param behavior is not desired, override the method to customize. Example:

class Example < ActiveRecord::Base
  def to_param
    # Custom implementation

The #to_param and .tokener_find methods complement each other in this regard.

Class Helpers

There is a convenience method for rebuilding/upgrading database records with token support. By default it will generate tokens for records with missing tokens. Example:


Use force: true to force all tokens to be rebuilt whether they exist or not. Example:

Song.rebuild_tokens! force: true

TIP: Tokens are rebuilt using the transformer as defined by the tokener macro. Use a custom transformer to customize.


For convenience, a Rails migration generator is available from the command line in case it is necessary to add token support to previously created tables.


rails generate tokener:migrate table [column] [options]


rails generate tokener:migrate songs

This will then yield the following migration:

db/migrate/<datetime stamp>_add_token_to_songs.rb


To test, run:

bundle exec rake


Read Semantic Versioning for details. Briefly, it means:

  • Major (X.y.z) - Incremented for any backwards incompatible public API changes.
  • Minor (x.Y.z) - Incremented for new, backwards compatible, public API enhancements/fixes.
  • Patch (x.y.Z) - Incremented for small, backwards compatible, bug fixes.

Code of Conduct

Please note that this project is released with a CODE OF CONDUCT. By participating in this project you agree to abide by its terms.


Read CONTRIBUTING for details.


Copyright 2010 Alchemists. Read LICENSE for details.


Read CHANGES for details. Built with Gemsmith.


Developed by Brooke Kuhlmann at Alchemists.