Master files should sync to local storage

[davidschober]

Description

Master files in Avalon should sync to local storage during the pilot phase. We have not made a decision about long term storage of files, and it is out of scope for this project. For now we want to have a copy of renamed master files (not derivatives) synced to local storage.

Done looks like

• master files are stored locally
• Decision is made whether to keep files on S3

[davidschober]

Note, we decided this might be able to be handled manually during the pilot.

[ccaizzi]

@davidschober new masters will be stored on S3 until this get figured out. old masters will remain on Isilon, correct? also, let's give this a deadline of July 15.

[davidschober]

Yes. We may sync occasionally. Masters are safe in s3

…

Sent from my iPhone

On Jun 5, 2017, at 1:56 PM, ccaizzi ***@***.***> wrote: @davidschober new masters will be stored on S3 until this get figured out. old masters will remain on Isilon, correct? also, let's give this a deadline of July 15. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ccaizzi]

I'll be interested in seeing how much is costs to sync the masters back down.

[ccaizzi]

@davidschober let's schedule a meeting with @mbklein to align this with our preservation strategy.

[davidschober]

@mbklein and @Toputnal and @d-venckus let's discuss our sync strategy this week. Michael, this is something Jim and David can setup a local worker to sync from the s3 nightly correct? Using using something like http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

[Toputnal]

I'm currently syncing masterfiles to my local, mirrored disks on my desktop. I will request a new isilon share and will move the files from my desktop to the new isilon share so as to avoid downloading everything from amazon more than once.

[Toputnal]

I am copying the data from my desktop to the Isilon on farchive21 via avalonweb1p. After the initial copy. I will look into setting up a nightly cron job on avalonweb1p to just pull down diffs from s3.

[Toputnal]

I have setup a cron job on avalonweb1p (our "offline" local Avalon web instance) to sync/pull down avalon-masterfiles-cfx3wj9 and put the contents in farchive21. The job runs at 2:22 AM each day. @mbklein @carrickr is this the only s3 bucket that needs to be sync'ed locally?

[davidschober]

Thanks @Toputnal . I believe that's correct. @carrickr can you confirm. We're only concerned with syncing master files that have been ingested.

[carrickr]

@davidschober @Toputnal

That's correct. Currently the only bucket we care about in production.

[davidschober]

@Toputnal I'm going to close this as complete. I will write docs about where things are syncing to.