Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not upload files with identical etags #536

Closed
wants to merge 1 commit into from
Closed
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.
+26 −1
Diff settings

Always

Just for now

@@ -63,18 +63,25 @@ def run
files += set
end
keys_to_destroy = files.all.map(&:key)
etags = read_etags(files)

# Upload all the files in the output folder to the clouds
puts 'Uploading local files'
FileUtils.cd(src) do
files = Dir['**/*'].select { |f| File.file?(f) }
files.each do |file_path|
key = path ? File.join(path, file_path) : file_path

keys_to_destroy.delete(key)

local_etag = calc_local_etag(file_path)
remote_etag = etags[key]
next if remote_etag && remote_etag == local_etag

This comment has been minimized.

Copy link
@mpapis

mpapis Apr 19, 2015

Member

avoid calculating local hash if remote is not available:

remote_etag = etags[key]
if remote_etag
  local_etag = calc_local_etag(file_path)
  next if remote_etag == local_etag
end

This comment has been minimized.

Copy link
@ddfreyne

ddfreyne May 31, 2015

Author Member

Thanks, updated! (see #552—not this PR)

directory.files.create(
key: key,
body: File.open(file_path),
public: true)
keys_to_destroy.delete(key)
end
end

@@ -89,6 +96,24 @@ def run

private

def read_etags(files)
case config[:provider]
when 'aws'
files.each_with_object({}) do |file, etags|
etags[file.key] = file.etag
end
else
{}
end
end

def calc_local_etag(file_path)
case config[:provider]
when 'aws'
Digest::MD5.file(file_path).hexdigest
end
end

# Prints the given message on stderr and exits.
def error(msg)
raise RuntimeError.new(msg)
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.