New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Purge variants with their blobs #31319
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -19,7 +19,7 @@ def initialize(path:, storage_account_name:, storage_access_key:, container:) | |
end | ||
|
||
def upload(key, io, checksum: nil) | ||
instrument :upload, key, checksum: checksum do | ||
instrument :upload, key: key, checksum: checksum do | ||
begin | ||
blobs.create_block_blob(container, key, io, content_md5: checksum) | ||
rescue Azure::Core::Http::HTTPError | ||
|
@@ -30,19 +30,19 @@ def upload(key, io, checksum: nil) | |
|
||
def download(key, &block) | ||
if block_given? | ||
instrument :streaming_download, key do | ||
instrument :streaming_download, key: key do | ||
stream(key, &block) | ||
end | ||
else | ||
instrument :download, key do | ||
instrument :download, key: key do | ||
_, io = blobs.get_blob(container, key) | ||
io.force_encoding(Encoding::BINARY) | ||
end | ||
end | ||
end | ||
|
||
def delete(key) | ||
instrument :delete, key do | ||
instrument :delete, key: key do | ||
begin | ||
blobs.delete_blob(container, key) | ||
rescue Azure::Core::Http::HTTPError | ||
|
@@ -51,16 +51,32 @@ def delete(key) | |
end | ||
end | ||
|
||
def delete_prefixed(prefix) | ||
instrument :delete_all, prefix: prefix do | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed the instrumentation name to match the others in: 7609ca0 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Gah, thanks! |
||
marker = nil | ||
|
||
loop do | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could this potentially block forever? Or does There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can provide the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍, the |
||
results = blobs.list_blobs(container, prefix: prefix, marker: marker) | ||
|
||
results.each do |blob| | ||
blobs.delete_blob(container, blob.name) | ||
end | ||
|
||
break unless marker = results.continuation_token.presence | ||
end | ||
end | ||
end | ||
|
||
def exist?(key) | ||
instrument :exist, key do |payload| | ||
instrument :exist, key: key do |payload| | ||
answer = blob_for(key).present? | ||
payload[:exist] = answer | ||
answer | ||
end | ||
end | ||
|
||
def url(key, expires_in:, filename:, disposition:, content_type:) | ||
instrument :url, key do |payload| | ||
instrument :url, key: key do |payload| | ||
base_url = url_for(key) | ||
generated_url = signer.signed_uri( | ||
URI(base_url), false, | ||
|
@@ -77,7 +93,7 @@ def url(key, expires_in:, filename:, disposition:, content_type:) | |
end | ||
|
||
def url_for_direct_upload(key, expires_in:, content_type:, content_length:, checksum:) | ||
instrument :url, key do |payload| | ||
instrument :url, key: key do |payload| | ||
base_url = url_for(key) | ||
generated_url = signer.signed_uri(URI(base_url), false, permissions: "rw", | ||
expiry: format_expiry(expires_in)).to_s | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,30 +16,30 @@ def initialize(root:) | |
end | ||
|
||
def upload(key, io, checksum: nil) | ||
instrument :upload, key, checksum: checksum do | ||
instrument :upload, key: key, checksum: checksum do | ||
IO.copy_stream(io, make_path_for(key)) | ||
ensure_integrity_of(key, checksum) if checksum | ||
end | ||
end | ||
|
||
def download(key) | ||
if block_given? | ||
instrument :streaming_download, key do | ||
instrument :streaming_download, key: key do | ||
File.open(path_for(key), "rb") do |file| | ||
while data = file.read(64.kilobytes) | ||
yield data | ||
end | ||
end | ||
end | ||
else | ||
instrument :download, key do | ||
instrument :download, key: key do | ||
File.binread path_for(key) | ||
end | ||
end | ||
end | ||
|
||
def delete(key) | ||
instrument :delete, key do | ||
instrument :delete, key: key do | ||
begin | ||
File.delete path_for(key) | ||
rescue Errno::ENOENT | ||
|
@@ -48,16 +48,24 @@ def delete(key) | |
end | ||
end | ||
|
||
def delete_prefixed(prefix) | ||
instrument :delete_prefixed, prefix: prefix do | ||
Dir.glob(path_for("#{prefix}*")).each do |path| | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I was under the impression it would only do so if we set the irb(main):006:0> Dir.glob("/Users/george/Desktop/*")
=> []
irb(main):007:0> Dir.glob("/Users/george/Desktop/*", File::Constants::FNM_DOTMATCH)
=> ["/Users/george/Desktop/.", "/Users/george/Desktop/..", "/Users/george/Desktop/.DS_Store", "/Users/george/Desktop/.localized"] Is that platform-dependent?
In Active Storage’s own usage, yes.
All objects are deleted. That seems correct, though perhaps not desirable. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Not sure. It could be Ruby version dependent. I definitely remember seeing it in our app on Ruby 2.2.
I meant to say: if There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I’m wrong. Active Storage’s usage of
I’m not seeing it on Ruby 2.2: irb(main):002:0> RUBY_VERSION
=> "2.2.8"
irb(main):003:0> Dir.glob("/Users/george/Desktop/*")
=> []
irb(main):004:0> Dir.glob("/Users/george/Desktop/*", File::Constants::FNM_DOTMATCH)
=> ["/Users/george/Desktop/.", "/Users/george/Desktop/..", "/Users/george/Desktop/.DS_Store", "/Users/george/Desktop/.localized"]
If they’re returned by There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, I was thinking of |
||
FileUtils.rm_rf(path) | ||
end | ||
end | ||
end | ||
|
||
def exist?(key) | ||
instrument :exist, key do |payload| | ||
instrument :exist, key: key do |payload| | ||
answer = File.exist? path_for(key) | ||
payload[:exist] = answer | ||
answer | ||
end | ||
end | ||
|
||
def url(key, expires_in:, filename:, disposition:, content_type:) | ||
instrument :url, key do |payload| | ||
instrument :url, key: key do |payload| | ||
verified_key_with_expiration = ActiveStorage.verifier.generate(key, expires_in: expires_in, purpose: :blob_key) | ||
|
||
generated_url = | ||
|
@@ -77,7 +85,7 @@ def url(key, expires_in:, filename:, disposition:, content_type:) | |
end | ||
|
||
def url_for_direct_upload(key, expires_in:, content_type:, content_length:, checksum:) | ||
instrument :url, key do |payload| | ||
instrument :url, key: key do |payload| | ||
verified_token_with_expiration = ActiveStorage.verifier.generate( | ||
{ | ||
key: key, | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,7 +14,7 @@ def initialize(**config) | |
end | ||
|
||
def upload(key, io, checksum: nil) | ||
instrument :upload, key, checksum: checksum do | ||
instrument :upload, key: key, checksum: checksum do | ||
begin | ||
bucket.create_file(io, key, md5: checksum) | ||
rescue Google::Cloud::InvalidArgumentError | ||
|
@@ -25,7 +25,7 @@ def upload(key, io, checksum: nil) | |
|
||
# FIXME: Download in chunks when given a block. | ||
def download(key) | ||
instrument :download, key do | ||
instrument :download, key: key do | ||
io = file_for(key).download | ||
io.rewind | ||
|
||
|
@@ -38,7 +38,7 @@ def download(key) | |
end | ||
|
||
def delete(key) | ||
instrument :delete, key do | ||
instrument :delete, key: key do | ||
begin | ||
file_for(key).delete | ||
rescue Google::Cloud::NotFoundError | ||
|
@@ -47,16 +47,22 @@ def delete(key) | |
end | ||
end | ||
|
||
def delete_prefixed(prefix) | ||
instrument :delete_prefixed, prefix: prefix do | ||
bucket.files(prefix: prefix).all(&:delete) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Feels strange that the |
||
end | ||
end | ||
|
||
def exist?(key) | ||
instrument :exist, key do |payload| | ||
instrument :exist, key: key do |payload| | ||
answer = file_for(key).exists? | ||
payload[:exist] = answer | ||
answer | ||
end | ||
end | ||
|
||
def url(key, expires_in:, filename:, content_type:, disposition:) | ||
instrument :url, key do |payload| | ||
instrument :url, key: key do |payload| | ||
generated_url = file_for(key).signed_url expires: expires_in, query: { | ||
"response-content-disposition" => content_disposition_with(type: disposition, filename: filename), | ||
"response-content-type" => content_type | ||
|
@@ -69,7 +75,7 @@ def url(key, expires_in:, filename:, content_type:, disposition:) | |
end | ||
|
||
def url_for_direct_upload(key, expires_in:, content_type:, content_length:, checksum:) | ||
instrument :url, key do |payload| | ||
instrument :url, key: key do |payload| | ||
generated_url = bucket.signed_url key, method: "PUT", expires: expires_in, | ||
content_type: content_type, content_md5: checksum | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we only want this for
image?
variants? Isn't it a little sneaky that thedelete
behavior is different depending on the blob type?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Non-image blobs don’t have variants. Skipping this saves an unnecessary API call.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, right 👍