Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix file cache store delete_matched to work on keys longer than allowed filename size #49694

Merged
merged 1 commit into from Oct 19, 2023

Conversation

fatkodima
Copy link
Member

Fixes #49690.

For file cache store we url-encode keys to not to clash to with the file system path separators, like /. When the resulting key is longer than the FS max file name length, we split it into parts. But this splitting can be done incorrectly and made in-between the encode-sequence, while it should be done only on the boundaries.

Example: for key foo%20bar splitting into ['foo%2', '0bar'] is incorrect, should be ['foo', '%20bar'] or ['foo%20', 'bar'].

Then when we use delete_matched, we traverse the FS directories and try to decode dir names and get an error if we previously made a split in the incorrect place.

Copy link
Member

@jonathanhefner jonathanhefner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rindex with a regexp is a bit slower:

Benchmark script
# frozen_string_literal: true

require "benchmark/ips"
require "active_support/all"

FILENAME_MAX_SIZE = 226
FILENAME = "x" * (FILENAME_MAX_SIZE + 1)

Benchmark.ips do |x|
  x.report("original") do
    fname = FILENAME
    fname_paths = []
    begin
      fname_paths << fname[0, FILENAME_MAX_SIZE]
      fname = fname[FILENAME_MAX_SIZE..-1]
    end until fname.blank?
  end

  x.report("rindex") do
    fname = FILENAME
    fname_paths = []
    loop do
      if fname.length < FILENAME_MAX_SIZE
        fname_paths << fname
        break
      else
        index = fname.rindex(/%\h\h/, FILENAME_MAX_SIZE) || FILENAME_MAX_SIZE
        fname_paths << fname[0, index]
        fname = fname[index..-1]
      end
    end
  end

  x.compare!
end
Warming up --------------------------------------
            original    62.853k i/100ms
              rindex    42.464k i/100ms
Calculating -------------------------------------
            original    626.351k (± 2.0%) i/s -      3.143M in   5.019546s
              rindex    428.920k (± 1.2%) i/s -      2.166M in   5.049889s

Comparison:
            original:   626351.3 i/s
              rindex:   428919.5 i/s - 1.46x  slower

It also changes the cache keys, which might be an issue for existing deploys.

But, digging further, I think there is actually another bug in delete_matched. Consider the case where the matcher pattern spans the file separator boundary:

cache.write("A" * FILENAME_MAX_SIZE + "B", "value")
cache.delete_matched(/AB/)

The file path will be "A...A/B" so /AB/ will not match it.

But we can fix both that bug and the decode issue by removing the file separators from the key. I've pushed a commit with an updated test to cover both cases. Let me know what you think.

…ndaries

Co-authored-by: Jonathan Hefner <jonathan@hefner.pro>
@fatkodima
Copy link
Member Author

You are right! Squashed commits into one and added you as a co-author (author actually 😅)

@fatkodima fatkodima changed the title Fix file cache store to split url-encoded keys on encode-sequence boundaries Fix file cache store delete_matched to work on keys longer than allowed filename size Oct 19, 2023
@jonathanhefner jonathanhefner merged commit 9f6b721 into rails:main Oct 19, 2023
4 checks passed
@jonathanhefner
Copy link
Member

jonathanhefner commented Oct 19, 2023

Thank you, @fatkodima! 📂

Backported to 7-1-stable in 86104e0 and 7-0-stable in f630ca2.

@fatkodima fatkodima deleted the fix-file_store-key-splitting branch October 19, 2023 18:49
jonathanhefner added a commit that referenced this pull request Oct 19, 2023
Fix file cache store `delete_matched` to work on keys longer than allowed filename size

(cherry picked from commit 9f6b721)
jonathanhefner added a commit that referenced this pull request Oct 19, 2023
Fix file cache store `delete_matched` to work on keys longer than allowed filename size

(cherry picked from commit 9f6b721)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rails.cache.delete_matched throws invalid %-encoding for long url keys
2 participants