Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add key_hash for smaller index #135

Merged
merged 1 commit into from Jan 24, 2024
Merged

Add key_hash for smaller index #135

merged 1 commit into from Jan 24, 2024

Conversation

djmb
Copy link
Collaborator

@djmb djmb commented Jan 24, 2024

This adds a key_hash column to the solid_cache_entries table. This column is a 64-bit integer that is a truncated SHA256 hash of the key. This allows us to use a smaller index than the 1024 byte key column.

The byte_size column is also added to the table. This is the size of the value in bytes. This is for a separate feature but is included in this commit to piggy back on the migrations.

The migrations themselves are split into three steps. This allows existing tables to be migrated without downtime.

There is a new configuration field key_hash_stage that can be set to :ignored, :unindexed or :indexed. This controls how the key_hash column is used and should be set as the migrations are completed.

The default is :indexed, which is what is required for fresh installations of solid_cache. In this case you can apply all three migrations at the same time.

In the case of an existing setup however, the steps are:

  1. Install the new gem, install the migrations and set config.solid_cache.key_hash_stage = :ignored
  2. Run the first migration, which adds the new columns
  3. Update config.solid_cache.key_hash_stage = :unindexed
  4. Backfill or truncate the table
  5. Run the second migration, which adds the indexes and null constraints
  6. Update config.solid_cache.key_hash_stage = :indexed
  7. Run the third migration, which removes the old key index

For later versions of the gem we'll assume that the key_hash column exists and the config.solid_cache.key_hash_stage. We'll also squash the migrations into a single file.

@djmb djmb force-pushed the key-hash branch 5 times, most recently from 5acdf16 to 545a980 Compare January 24, 2024 11:37
This adds a key_hash column to the solid_cache_entries table. This
column is a 64-bit integer that is a truncated SHA256 hash of the key.
This allows us to use a smaller index than the 1024 byte key column.

The byte_size column is also added to the table. This is the size of
the value in bytes. This is for a separate feature but is included in
this commit to piggy back on the migrations.

The migrations themselves are split into three steps. This allows
existing tables to be migrated without downtime.

There is a new configuration field `key_hash_stage` that can be set to
`:ignored`, `:unindexed` or `:indexed`. This controls how the key_hash
column is used and should be set as the migrations are completed.

The default is :indexed, which is what is required for fresh
installations of solid_cache. In this case you can apply all three
migrations at the same time.

In the case of an existing setup however, the steps are:

1. Install the new gem, install the migrations and set
   `config.solid_cache.key_hash_stage = :ignored`
2. Run the first migration, which adds the new columns
3. Update `config.solid_cache.key_hash_stage = :unindexed`
4. Backfill or truncate the table
5. Run the second migration, which adds the indexes and null constraints
6. Update `config.solid_cache.key_hash_stage = :indexed`
7. Run the third migration, which removes the old key index

For later versions of the gem we'll assume that the key_hash column
exists and the `config.solid_cache.key_hash_stage`. We'll also squash
the migrations into a single file.
@djmb djmb merged commit c933b9c into main Jan 24, 2024
34 checks passed
@djmb djmb deleted the key-hash branch January 24, 2024 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant