Investigate farchive/ Files moved. Isilon report seems low

[davidschober]

DONE

Give a report to CC about files moved, data in .

It seems for the last quarter we only increased our storage by ~2tb. Which seems considerably lower than it should be. @mbklein to investigate.

@d-venckus can you do a report on our isilon storage and attach to this ticket?

[davidschober]

Thanks, MBK.

Kurt said: 1306 request for items of which 745 were closes so very likely already in AVR​.

A request is per title for video and he said they do not do full albums for audio requests.

So that is 561 new videos and audio. Without knowing the split, this may be tough to assess, but I am worried about the AVR copies to Farchive during this time.

Besides over 2TB added from one collection to a DL share, we did add multiple image collections to repo during this time which should have made it over to farchive as well. Jen, Nicole, can you verify that we did not add any special collections to AVR during this time?

From: Michael B. Klein
Sent: Thursday, January 19, 2017 4:06 PM
To: Carolyn Caizzi
Subject: Re: dates for the backups and the 2 TB change

OK, I’ll go over this with David soon.

mbk

From: Carolyn Caizzi carolyn.caizzi@northwestern.edu
Date: Thursday, January 19, 2017 at 2:53 PM
To: "Michael B. Klein" michael.klein@northwestern.edu
Subject: Fw: dates for the backups and the 2 TB change

I forwarded this to David already, but this does seem low for us considering we added one collection to DL that was over 2TB--we definitely added others...I am worried that those issues we had mean that videos were not being moved to farchive for AVR.
From: Donald R Hammer
Sent: Friday, January 13, 2017 5:28 PM
To: Carolyn Caizzi; Robert Stephen Trautvetter
Subject: RE: dates for the backups and the 2 TB change

Carolyn,

Attached is the output from Client backup of Isilon1.

Don

From: Carolyn Caizzi
Sent: Friday, January 13, 2017 4:42 PM
To: Donald R Hammer; Robert Stephen Trautvetter
Subject: dates for the backups and the 2 TB change

Hi Don,

I am slightly puzzled at that 2TB increase over 3 month period. Could you give me the dates for when the first set of tapes completed and then when you ran the 2nd backup? I want to then have our devops run a report like they have been doing to make sure nothing has gone awry. My content folks told me they definitely uploaded some large collections in the fall that should amount to over 2 TB.

Thanks,
Carolyn

Carolyn Caizzi
Repository and Digital Curation, Head
Northwestern University Libraries
Northwestern University
www.library.northwestern.edu
carolyn.caizzi@northwestern.edu
847.467.3898

[davidschober]

@mbklein will run an audit of files to ensure that master files exist in Farchive.

We will report a csv of Failures.

[ccaizzi]

@mbklein have you had a chance to work through this?

[ccaizzi]

below from @mbklein. For the 1800+ files, he will generate a report for RDC DCM team to analyze.

Hi Carolyn,

Avalon automatically moves MasterFiles to /farchive after successful ingest and transcoding.

I’ve taken several passes throught the MasterFiles in Avalon and the files that remain in Avalon’s ingest/dropbox directories. As far as I can tell, all video files that should have been moved to farchived were moved either by Avalon or by my script the last time we had an Isilon quota issue.

The files that remain in the dropbox fall into the following categories:

1.   1,854 audio files that my previous script missed (.aiff, .wav., .mp3, .m4a), that will have to be status-checked to make sure they’ve been properly ingested before I move them. (A significant number of these files are part of failed batch ingests, so not all of them can be considered “missing” from farchive.)
   

2.   18 obvious test items (e.g., collections named “Streaming Testing” or copies of the Avalon demo videos).
   

3.   5 other videos that I have been unable to match to any existing AVR MasterFile objects.
   

I have been unable to come up with an explanation for the before/after size anomaly in the backup report, but I am confident (based on my spot checks on December 7 and my current investigation) that all files that needed to be moved in December were moved either to farchive19 or farchive20. Those files that remain behind can be taken care of quickly once I finish their status checks, and do not represent a particularly significant amount of data in terms of file size (because they are audio).

Michael