Python
Clone or download
Permalink
Failed to load latest commit information.
docs docs/setup - version bump Aug 9, 2018
legacy_scripts move premis back to main folder Apr 2, 2018
26_XYZ-22_Rec709.cube Updated LUT for chromatic adaptation Aug 2, 2018
LICENSE.txt changes License so that IFI is credited Apr 10, 2018
MANIFEST.in adds MANIFEST.in to include the LUT Jul 19, 2018
Objects.py Objects/dfxml - actually adds sha512 support Feb 12, 2018
README.rst README.rst - larger heading Jul 13, 2018
accession.py accessioning - allows user to specify depositor reference number May 26, 2018
accession_register.py adds accession_register.py minus a README entry Jun 6, 2018
as11fixity.py as11fixity - PEP-08 cleanup Sep 9, 2017
batchaccession.py batchaccession - display filmographic titles during QC phase Jun 28, 2018
batchfixity.py batchfixity/batchmetadata - PEP08 compliance Jul 12, 2017
batchmetadata.py batchfixity/batchmetadata - PEP08 compliance Jul 12, 2017
batchrename.py batchrename - PEP08 compliance Jul 12, 2017
bitc.py PEP-08 compliance Aug 19, 2017
concat.py fixes OE number input check - fixes #247 Apr 6, 2018
copyit.py copyit - allows user to continue when not enough space - useful when … Jul 6, 2018
db_cleaning.md db_cleaning.md Oct 9, 2017
db_cleaning.py brings up to python3 Jun 21, 2018
dcpaccess.py dcpaccess.py - fixes missing bracket Jul 19, 2018
dcpfixity.py dcpfixity - dont force bagit Apr 19, 2017
dcpsubs2srt.py dcpsubs2srt - PEP08 compliance Jul 12, 2017
deletefiles.py adds file deletion script that deletes files and updates logs and man… Mar 19, 2018
dfxml.py Objects/dfxml - actually adds sha512 support Feb 12, 2018
dpxonly.py dpxonly - adds check for spaces Apr 11, 2017
durationcheck.py Duration check - adds mkv to search Jul 13, 2017
dvsip.py brings up to python3 Jun 21, 2018
fakexdcam.py fakexdcam - changes permissions Mar 16, 2018
ffv1mkvvalidate.py ffv1mkvvalidate - adds argparse Aug 1, 2017
ififuncs.py ififuncs - fixes diff_framemd5s() bug/typo Aug 9, 2018
loopline.py loopline - bitc runs first, adds proxies dir, adds dvsip Mar 18, 2017
loopline_repackage.py loopline_repackage - adds filmographic reference number to logfile Jun 25, 2018
makedfxml.py makedfxml - adds no hash argument Apr 17, 2018
makedpx.py makedpx - puts general csv report into desktop subfolder, fies#209 Aug 10, 2017
makeffv1.py makeffv1/dvsip - fixes DV detection for reals this time Jun 6, 2017
makefolders.py makefolders - fixes typos Nov 15, 2017
makepbcore.py makepbcore - check the stream, not the container section for interlan… Aug 10, 2018
makeuuid.py PEP-08 compliance Aug 19, 2017
manifest.py manifest - adds algorithm type to log Feb 4, 2018
masscopy.py PEP-08 compliance Aug 19, 2017
massqc.py PEP-08 compliance Aug 19, 2017
mezzaninecheck.py PEP-08 compliance Aug 19, 2017
multicopy.py adds multicopy.py Apr 26, 2018
normalise.py normalise - puts framemd5s in the metadata dir - thx @raecasey Aug 9, 2018
order.py order/ififuncs - get parent of accessioned package May 9, 2018
package_update.py package_update - completes rename from rearrange.py Jul 23, 2018
packagecheck.py adds packagecheck script - verify IFI SIP/AIP folder structures Apr 2, 2018
playerseq.py playerseq.py - adds workflows specific image sequence script Oct 27, 2016
premis.py move premis back to main folder Apr 2, 2018
premis_agents.csv PREMISagents, adds Bertrand caron Feb 19, 2018
premisgui.ui premisgui - moves around some elements and adds multi select to grade Dec 4, 2016
prores.py permissions Aug 17, 2016
python_cheat_sheet.py permissions Aug 17, 2016
qctools.py qctools - recursive loop and audio channel check Apr 7, 2017
rawbatch.py manifest name change Jan 10, 2017
rearrange.py rearrange.py - make executable Apr 3, 2018
seq.py adding comments to the fps alternative argument Jan 13, 2017
seq2dv.py seq2dv - adds hack to stop getting fooled by hidden files Dec 21, 2016
seq2ffv1.py seq2ffv1 - uses UUID for output filename/logs/metadata Sep 8, 2017
seq2mezz.py updating suggestions by KOL Jun 19, 2017
seq2prores.py premis scripts - adds intellectual entity objects Jan 4, 2017
setup.py Merge branch 'master' into normalise Aug 9, 2018
sha512deep.py sha512deep - some more PEP-8 compliance Jul 1, 2017
sipcreator.py sipcreator - adds username to package_update command Jul 23, 2018
strongbox_fixity.py strongbox_fixity - adds option to check differences between strongbox… May 16, 2018
testfiles.py testfiles - adss argparse and docstring Jul 14, 2017
update.py update - checks out master branch May 15, 2017
validate.py validate - adds docstrings and PEP-08 cleanup Apr 26, 2018
viruscheck.py viruscheck - adds verbose option, PEP-08 cleanup Apr 19, 2018
walk_to_dfxml.py walk_to_dfxml - only uses md5and sha512 for real this time Feb 13, 2018

README.rst

Full documentation at: http://ifiscripts.readthedocs.io/en/latest/index.html

http://readthedocs.org/projects/ifiscripts/badge/?version=latest

Introduction

Summary

These are Python 2.7 scripts that facilitate collections management workflows within the IFI Irish Film Archive. These scripts have been tested in OSX, Windows 7 & 10, Ubuntu 14.04 & 16.04. They are located here on github: https://github.com/kieranjol/IFIscripts

Most scripts take either a file or a directory as their input, for example makeffv1.py filename.mov or premis.py path/to/folder_of_stuff. (It's best to just drag and drop the folder or filename into the terminal as this provides the absolute path).

We want the project to be as reuseable as possible in different institutions and contexts. Some scripts, particularly anything to do with Object Entry or Accessioning will be quite IFI specific, but other scripts such as makeffv1.py, dcpaccess.py and many others have been used in a variety of contexts in several different countries.

The project uses the MIT license, and we encourage the reuse, modification and study of the scripts. It's always nice to hear when the scripts have been reused in some way, but it's not necessary to let us know.

Purpose

These python scripts facilitate much of our collections management procedures for digitised and born digital objects in the Irish Film Institute. We utilise a lot of open source tools, so we wanted to make these scripts as open as possible. This is why this project has the MIT License.

The Irish Film Institute has followed the SPECTRUM museum collections management standard for several years. These scripts attempt to follow SPECTRUM procedures while also utilising some of the concepts of the Open Archival Information System (OAIS). Initially the scripts only handled single video files, but they are now capable of handling:

  • Digital Cinema Packages
  • XDCAM cards
  • DPX/TIFF image sequences
  • Documents (.doc, .pdf etc)
  • Images (.jpg, .TIFF etc)

An example workflow might be:

  • A digital object is created or acquired by the IFI, and ingest begins.
  • sipcreator.py is run on the object. This script:
    • generates an Object Entry identifier (eg OE-1234)
    • creates a folder structure for logs, metadata, objects
    • generates a UUID, extracts technical metadata
    • generates a md5 checksum manifest
    • and see the usage section for more.
  • All of these preservation events are logged in a log file located in the logs directory. This log file tries to use PREMIS (PREservation Metadata Implementation Strategies) terminology as much as possible.
  • Even though the package has yet to be accessioned, temporary backups are required. copyit.py will generate backups, and it will use the checksum manifest generated by sipcreator.py to verify the integrity of the file transfer.
  • If the package contains FFV1 or Matroska files, perhaps ffv1mkvvalidate.py could run, which uses mediaconch to verify the compliance of the files, and stores the information in the logfile.
  • If the package passes our Quality Control Procedures, then it will be accessioned. accession.py will generate an accession number, rename the OE number with the accession number, generate a SHA-512 manifest and update the log file to document these new preservation events.
  • A large batch of items can be accessioned using batchaccession.py. If the -pbcore command line argument is used with the accessioning scripts, technical metadata based on the PBCore standard will be generated in CSV format. This process can be run seperately by using makepbcore.py. CSV was chosen instead of XML as this allows us to immediately import the CSV into our database system so that we have item level records.
  • Access copies may be needed, so low-res watermarked proxies can be generated with bitc.py, or high res mezzanines with prores.py.
  • The accessioned package can then be written to preservation storage, again using the copyit.py command.

So this is just one way of using the scripts from acquisition to preservation storage, but there are many other scripts for specific workflows, which you can investigate further down in the documentation.

Table of Contents

.. toctree::
   :maxdepth: 4
   :caption: Contents:

   installation
   contributing
   usage
   credits