Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Generates string tokens as alternative IDs for easy reference.
Ruby
branch: master
Failed to load latest commit information.
gemfiles Added Rails 4.2.x support.
lib Added version release notes.
spec Removed RSpec garbage collection support.
.gitignore Removed Gemfile.lock from .gitignore.
.ruby-version Updated to Ruby 2.2.2.
.travis.yml
CHANGELOG.md Added version release notes.
CODE_OF_CONDUCT.md Added code of conduct documentation. [ci skip]
CONTRIBUTING.md
Gemfile Switched from HTTP to HTTPS when sourcing from RubyGems. [ci skip]
Gemfile.lock Updated gem dependencies.
Guardfile
LICENSE.md Updated company links to be HTTPS instead of HTTP. [ci skip]
README.md
Rakefile Initial commit.
tokener.gemspec Added the terminal notifier gem.

README.md

Overview

Gem Version Code Climate GPA Code Climate Coverage Gemnasium Status Travis CI Status Gittip

Generates string tokens as alternative IDs for easy reference. This can prove useful for URL addresses, internal code lookup, human readability, file system files, etc.

Features

  • Supports ActiveRecord models.
  • Supports token generation from single/multiple attributes (including delimiter customization).
  • Supports custom token attribute naming. By default, the token attribute is used.
  • Supports token validation using the similar criteria to token generation.
  • Supports token scopes.
  • Supports token blacklists.
  • Supports custom token transforms. By default, tokens are transliterated, URI encoded, and URI tokenized. Additionally, tokens can be decoded and detokenized.
  • Supports custom unique token generation. By default, duplicate tokens are suffixed with a 32-character hex. Example: "-3885821dc01e3df51b10e08df5c007d1".
  • Supports finding records by token, primary key, or custom key.
  • Supports rebuilding of tokens either gracefully or forcefully.

Requirements

  1. Any of the following Ruby VMs:
  2. Ruby on Rails 4.x.x.

Setup

For a secure install, type the following from the command line (recommended):

gem cert --add <(curl -Ls https://www.alchemists.io/gem-public.pem)
gem install tokener --trust-policy MediumSecurity

NOTE: A HighSecurity trust policy would be best but MediumSecurity enables signed gem verification while allowing the installation of unsigned dependencies since they are beyond the scope of this gem.

For an insecure install, type the following (not recommended):

gem install tokener

Add the following to your Gemfile:

gem "tokener"

Usage

The are many options available to you, see below.

Single Attribute

Basic, out-of-the-box usage would be as follows:

class Song < ActiveRecord::Base
  tokener :label
end

The resulting attribute=value pairs would yield: {label: "Can't You Hear Me Knocking", token: "cant_you_hear_me_knocking"}.

Multiple Attributes

When a single attribute is not enough, use multiple attributes. Example (first_name = "Jimi" and last_name = "Hendrix"):

class Musician < ActiveRecord::Base
  tokener [:first_name, :last_name]
end

The resulting token would be: "jimi_hendrix".

Attribute order is important as the reverse is possible too:

class Musician < ActiveRecord::Base
  tokener [:last_name, :first_name]
end

The resulting token would be:: "hendrix_jimi".

Multiple Attribute Custom Delimiter

When the default delimiter does not suffix, it can be customized:

class Musician < ActiveRecord::Base
  tokener [:first_name, :last_name], delimiter: '-'
end

The resulting token would be: "jimi-hendrix".

Custom Token

If the default behavior is not to your liking then the ability to define a specific token attribute is possible. Example:

class Song < ActiveRecord::Base
  tokener :label, token: :slug
end

The resulting attribute/value pairs would be: {label: "Can't You Hear Me Knocking", slug: "cant_you_hear_me_knocking"}.

Validation

Token validation is performed prior to saving, using similar criteria used in creation. By default, this results in the following (example):

validates :token,
          presence: true,
          format: {
            with: /\A[0-9a-zA-Z\-\_\%]+\z/,
            message: "must consist of digits, lowercase letters, dashes, and/or underscores only."
          }

If scope is provided, then the following would be added (example):

uniqueness: {scope: [:category_id]}

If blacklist is provided, then the following would be added (example):

exclusion: ["one"]

NOTE: The capital letters and the percent (%) are allowed in validation to support URI encoded characters.

TIP: These are the validation settings, if enabled, by this gem. Should these settings not cover specific requirements, then the option of applying custom validations to your class can be used in addition to those provide by this gem.

Blacklists

Some situations might require the blacklisting of reserved tokens. This can be done by defining a blacklist as follows:

class Song < ActiveRecord::Base
  tokener :label, blacklist: %w(rocky_raccoon helter_skelter revolution_9)
end

Any generated token matching a blacklisted token would result in a validation error.

Scopes

Scopes allow token generation to be limited to single/multiple attributes. Example:

class Album < ActiveRecord::Base
  tokener :title, scope: :genre_id
end

This would limit the token to be unique for with the same genre. Based on the definition above, the following would be valid:

  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 1
  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 2

However, it would be invalid if the genre ID was 1 for both records.

More than one scope is also available by passing in an array to the scope. Example:

class Album < ActiveRecord::Base
  tokener :title, scope: [:genre_id, :category_id]
end

Based on the example above, the following would be valid:

  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 1, category_id: 1
  • title: "Houses of the Holy", token: "houses_of_the_holy", genre_id: 2, category_id: 1

Using more than one of the same genre_id and category_id combinations would throw a validation error.

Transforms

Transformation of raw input into a token is performed via the Tokener::Transformer::Default class. This class provides the default implementation for defining what unicode characters are allowed for tokenization, transliteration, and encoding. It also provides methods for detokenization and decoding. To customize, extend the Tokener::Transformer::Base abstract class and use the Tokener::Transformer::Default implementation as an example. Once a custom implementation has been built, it can be defined via the tokener macro. Example:

class Example < ActiveRecord::Base
  tokener :label, transformer: MyCustomTransformer.new
end

Uniqueness

By default, the ability to auto-generate unique tokens is disabled. When enabled, unique tokens are always generated after the first occurance. Example:

class Song < ActiveRecord::Base
  tokener :label, token: :slug, unique: true
end

The following tokens would be generated (with a unique, 32-character, hex applied after the first occurance). Example:

  • 1st Occurance: "cant_you_hear_me_knocking"
  • 2nd Occurance: "cant_you_hear_me_knocking-3885821dc01e3df51b10e08df5c007d1"
  • 3rd Occurance: "cant_you_hear_me_knocking-d2065aab58203c56440e8693e669933e"

Generation of unique suffixes can be customized by extending the Tokener::Uniquer::Base abstract class. Use the Tokener::Uniquer::Hex implementation as an example. When ready to use a custom uniquer, define it via the tokener macro as follows:

class Example < ActiveRecord::Base
  tokener :label, uniquer: MyCustomUniquer
end

Finders

The ability to search by token or ID is supported via the .find_by_identifier method. The following will yield the same result:

Song.find_by_identifier "cant_you_hear_me_knocking"
Song.find_by_identifier 1

The token is searched first, otherwise defaulting to the original ID.

It is recommended that the .find_by_identifier method be used when finding records as it makes finding encoded tokens easy. Example:

Song.find_by_identifier "你能听到我的敲门" # Finds the URI encoded version (i.e.
"%E4%BD%A0%E8%83%BD%E5%90%AC%E5%88%B0%E6%88%91%E7%9A%84%E6%95%B2%E9%97%A8") stored in the database.

Song.find_by_identifier "%E4%BD%A0%E8%83%BD%E5%90%AC%E5%88%B0%E6%88%91%E7%9A%84%E6%95%B2%E9%97%A8" # Accepts raw
encoding too.

Additionally, the default #to_param method is overwritten so if a token value exists, the token is returned (otherwise the original ID is returned). Example:

"cant_you_hear_me_knocking" # If token exists.

"你能听到我的敲门" # If token exists and is encoded (is decoded before returning).

1 # When no token exists, the primary ID is returned instead.

These makes generating pretty URLs easy. Example:

/songs/cant_you_hear_me_knocking
/songs/你能听到我的敲门

If the default, #to_param, behavior is not desired, override the method to customize. Example:

class Example < ActiveRecord::Base
  def to_param
    # Custom implementation
  end
end

Class Helpers

There is a convenience method for rebuilding/upgrading database records with token support. By default it will generate tokens for records with missing tokens. Example:

Song.rebuild_tokens!

Use force: true to force all tokens to be rebuilt whether they exist or not. Example:

Song.rebuild_tokens! force: true

TIP: Tokens are rebuilt using the transformer as defined by the tokener macro. Use a custom transformer to customize.

Generators

For convenience, a Rails migration generator is available from the command line in case it is necessary to add token support to previously created tables.

Usage:

rails generate tokener:migrate table [column] [options]

Example:

rails generate tokener:migrate songs

This will then yield the following migration:

db/migrate/<datetime stamp>_add_token_to_songs.rb

Tests

To test, run:

bundle exec rspec spec

Versioning

Read Semantic Versioning for details. Briefly, it means:

  • Patch (x.y.Z) - Incremented for small, backwards compatible bug fixes.
  • Minor (x.Y.z) - Incremented for new, backwards compatible public API enhancements and/or bug fixes.
  • Major (X.y.z) - Incremented for any backwards incompatible public API changes.

Code of Conduct

Please note that this project is released with a CODE OF CONDUCT. By participating in this project you agree to abide by its terms.

Contributions

Read CONTRIBUTING for details.

License

Copyright (c) 2010 Alchemists. Read the LICENSE for details.

History

Read the CHANGELOG for details. Built with Gemsmith.

Credits

Developed by Brooke Kuhlmann at Alchemists.

Something went wrong with that request. Please try again.