Generates string tokens as alternative IDs for easy reference.
Fetching latest commit…
Cannot retrieve the latest commit at this time.


Gem Version Code Climate GPA Code Climate Coverage Gemnasium Status Travis CI Status Patreon

Generates string tokens as alternative IDs for easy reference. This can prove useful for URL addresses, internal code lookup, human readability, file system files, etc.

Table of Contents


  • Supports ActiveRecord models.
  • Supports token generation from single/multiple attributes (including delimiter customization).
  • Supports custom token attribute naming. By default, the token attribute is used.
  • Supports token validation using the similar criteria to token generation.
  • Supports token scopes.
  • Supports token blacklists.
  • Supports custom token transforms. By default, tokens are transliterated, URI encoded, and URI tokenized. Additionally, tokens can be decoded and detokenized.
  • Supports custom unique token generation. By default, duplicate tokens are suffixed with a 32-character hex. Example: "-3885821dc01e3df51b10e08df5c007d1".
  • Supports finding records by token, primary key, or custom key.
  • Supports rebuilding of tokens either gracefully or forcefully.


  1. Ruby 2.x.x
  2. Ruby on Rails 5.x.x


For a secure install, type the following from the command line (recommended):

gem cert --add <(curl --location --silent
gem install tokener --trust-policy MediumSecurity

NOTE: A HighSecurity trust policy would be best but MediumSecurity enables signed gem verification while allowing the installation of unsigned dependencies since they are beyond the scope of this gem.

For an insecure install, type the following (not recommended):

gem install tokener

Add the following to your Gemfile:

gem "tokener"


The are many options available to you, see below.

Single Attribute

Basic, out-of-the-box usage would be as follows:

class Song < ActiveRecord::Base
  tokener :label

The resulting attribute=value pairs would yield: {label: "Can't You Hear Me Knocking", token: "cant_you_hear_me_knocking"}.

Multiple Attributes

When a single attribute is not enough, use multiple attributes. Example (first_name = "Jimi" and last_name = "Hendrix"):

class Musician < ActiveRecord::Base
  tokener [:first_name, :last_name]

The resulting token would be: "jimi_hendrix".

Attribute order is important as the reverse is possible too:

class Musician < ActiveRecord::Base
  tokener [:last_name, :first_name]

The resulting token would be:: "hendrix_jimi".

Multiple Attribute Custom Delimiter

When the default delimiter does not suffix, it can be customized:

class Musician < ActiveRecord::Base
  tokener [:first_name, :last_name], delimiter: '-'

The resulting token would be: "jimi-hendrix".

Custom Token

If the default behavior is not to your liking then the ability to define a specific token attribute is possible. Example:

class Song < ActiveRecord::Base
  tokener :label, token: :slug

The resulting attribute/value pairs would be: {label: "Can't You Hear Me Knocking", slug: "cant_you_hear_me_knocking"}.


Token validation is performed prior to saving, using similar criteria used in creation. By default, this results in the following (example):

validates :token,
          presence: true,
          format: {
            with: /\A[0-9a-zA-Z\-\_\%]+\z/,
            message: "must consist of digits, lowercase letters, dashes, and/or underscores only."

If scope is provided, then the following would be added (example):

uniqueness: {scope: [:category_id]}

If blacklist is provided, then the following would be added (example):

exclusion: ["one"]

NOTE: The capital letters and the percent (%) are allowed in validation to support URI encoded characters.

TIP: These are the validation settings, if enabled, by this gem. Should these settings not cover specific requirements, then the option of applying custom validations to your class can be used in addition to those provide by this gem.


Some situations might require the blacklisting of reserved tokens. This can be done by defining a blacklist as follows:

class Song < ActiveRecord::Base
  tokener :label, blacklist: %w(rocky_raccoon helter_skelter revolution_9)

Any generated token matching a blacklisted token would result in a validation error.


Scopes allow token generation to be limited to single/multiple attributes. Example:

class Album < ActiveRecord::Base
  tokener :title, scope: :genre_id

This would limit the token to be unique for with the same genre. Based on the definition above, the following would be valid:

  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 1
  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 2

However, it would be invalid if the genre ID was 1 for both records.

More than one scope is also available by passing in an array to the scope. Example:

class Album < ActiveRecord::Base
  tokener :title, scope: [:genre_id, :category_id]

Based on the example above, the following would be valid:

  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 1, category_id: 1
  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 2, category_id: 1

Using more than one of the same genre_id and category_id combinations would throw a validation error.


Transformation of raw input into a token is performed via the Tokener::Transformer::Default class. This class provides the default implementation for defining what unicode characters are allowed for tokenization, transliteration, and encoding. It also provides methods for detokenization and decoding. To customize, extend the Tokener::Transformer::Base abstract class and use the Tokener::Transformer::Default implementation as an example. Once a custom implementation has been built, it can be defined via the tokener macro. Example:

class Example < ActiveRecord::Base
  tokener :label, transformer:


By default, the ability to auto-generate unique tokens is disabled. When enabled, unique tokens are always generated after the first occurance. Example:

class Song < ActiveRecord::Base
  tokener :label, token: :slug, unique: true

The following tokens would be generated (with a unique, 32-character, hex applied after the first occurance). Example:

  • 1st Occurance: "cant_you_hear_me_knocking"
  • 2nd Occurance: "cant_you_hear_me_knocking-3885821dc01e3df51b10e08df5c007d1"
  • 3rd Occurance: "cant_you_hear_me_knocking-d2065aab58203c56440e8693e669933e"

Generation of unique suffixes can be customized by extending the Tokener::Uniquer::Base abstract class. Use the Tokener::Uniquer::Hex implementation as an example. When ready to use a custom uniquer, define it via the tokener macro as follows:

class Example < ActiveRecord::Base
  tokener :label, uniquer: MyCustomUniquer


The ability to search by token or ID is supported via the .find_by_identifier method. The following will yield the same result:

Song.find_by_identifier "cant_you_hear_me_knocking"
Song.find_by_identifier 1

The token is searched first, otherwise defaulting to the original ID.

It is recommended that the .find_by_identifier method be used when finding records as it makes finding encoded tokens easy. Example:

Song.find_by_identifier "你能听到我的敲门" # Finds the URI encoded version (i.e.
"%E4%BD%A0%E8%83%BD%E5%90%AC%E5%88%B0%E6%88%91%E7%9A%84%E6%95%B2%E9%97%A8") stored in the database.

Song.find_by_identifier "%E4%BD%A0%E8%83%BD%E5%90%AC%E5%88%B0%E6%88%91%E7%9A%84%E6%95%B2%E9%97%A8" # Accepts raw
encoding too.

Additionally, the default #to_param method is overwritten so if a token value exists, the token is returned (otherwise the original ID is returned). Example:

"cant_you_hear_me_knocking" # If token exists.

"你能听到我的敲门" # If token exists and is encoded (is decoded before returning).

1 # When no token exists, the primary ID is returned instead.

These makes generating pretty URLs easy. Example:


If the default, #to_param, behavior is not desired, override the method to customize. Example:

class Example < ActiveRecord::Base
  def to_param
    # Custom implementation

Class Helpers

There is a convenience method for rebuilding/upgrading database records with token support. By default it will generate tokens for records with missing tokens. Example:


Use force: true to force all tokens to be rebuilt whether they exist or not. Example:

Song.rebuild_tokens! force: true

TIP: Tokens are rebuilt using the transformer as defined by the tokener macro. Use a custom transformer to customize.


For convenience, a Rails migration generator is available from the command line in case it is necessary to add token support to previously created tables.


rails generate tokener:migrate table [column] [options]


rails generate tokener:migrate songs

This will then yield the following migration:

db/migrate/<datetime stamp>_add_token_to_songs.rb


To test, run:

bundle exec rake


Read Semantic Versioning for details. Briefly, it means:

  • Patch (x.y.Z) - Incremented for small, backwards compatible, bug fixes.
  • Minor (x.Y.z) - Incremented for new, backwards compatible, public API enhancements/fixes.
  • Major (X.y.z) - Incremented for any backwards incompatible public API changes.

Code of Conduct

Please note that this project is released with a CODE OF CONDUCT. By participating in this project you agree to abide by its terms.


Read CONTRIBUTING for details.


Copyright (c) 2010 Alchemists. Read LICENSE for details.


Read CHANGES for details. Built with Gemsmith.


Developed by Brooke Kuhlmann at Alchemists.