NFA Imports

My experience with NFA (Narodní Filmový Archiv) imports.

Using Lindat Repository Imports tools. Used ufal-point-dev (10.10.24.211) machine:

ssh -J kuchtiak@dijkstra.ms.mff.cuni.cz 10.10.24.211

NFA Import Example: Mizející řemesla

1. I've cloned the Lindat Repository Imports first
2. updated fixed_subject list in lindat-repository-imports/NFA/transformations/transform.xslt file:

    <xsl:variable name="fixed_subject">
        <xsl:choose>
            <xsl:when test="$collection='MNICHOV'">Mnichovská dohoda</xsl:when>
            <xsl:when test="$collection='GALERIE'">Galerie osobností</xsl:when>
            <xsl:when test="$collection='HEYDRICH'">Heydrichiáda</xsl:when>
            <xsl:when test="$collection='VZNIK_CSR'">Vznik ČSR</xsl:when>
            <xsl:when test="$collection='VYZNAMNE_POHRBY'">Významné pohřby</xsl:when>
            <xsl:when test="$collection='KURATORIUM'">Kuratorium</xsl:when>
            <xsl:when test="$collection='NARODNI_DIVADLO'">Národní divadlo</xsl:when>
            <xsl:when test="$collection='USPORNA_OPATRENI'">Úsporná opatření</xsl:when>
            <xsl:when test="$collection='ZDRAVOTNI_SOCIALNI'">Zdravotní a sociální péče</xsl:when>
            <xsl:when test="$collection='STAVEBNI_RUCH'">Pražský stavební ruch</xsl:when>
            <xsl:when test="$collection='NARODNI_PAMET'">Národní paměť</xsl:when>
            <xsl:when test="$collection='KVETEN_1945'">Květen 1945</xsl:when>
            <xsl:when test="$collection='MIZEJICI_REMESLA'">Mizející řemesla</xsl:when>
        </xsl:choose>
    </xsl:variable>

3. Original NFA format provides the list of DATA and METADATA in the following format (DATA and METADATA folders):

   cd ~/NFA/BK_2025_MIZEJICI_REMESLA
   ls DATA
      ZF0169_ID_3900243  ZF0262_ID_3900005  ZF0362_ID_3900425  ZF0411_ID_3900452  ZF0649_ID_3900674 ...
   # each DATA folder contains two files - full video, and short preview
   ls DATA/ZF0169_ID_3900243
      3900243_3_hd-master.mov  3900243_3_NAHLED.mp4
   # the METADATA folder contains metadata files for corresponding items
   ls METADATA
      3900243.xml  3900005.xml  3900245.xml  3900425.xml  3900452.xml  3900674.xml ... (contain metadata)

4. Copy each metadata (xml) file from METADATA folder to corresponding DATA folder:

   cd ~/NFA/BK_2025_MIZEJICI_REMESLA 
   for i in `ls ./METADATA/390*`; do f=$(basename $i .xml); cp -v $i DATA/*$f*; done

5. created following exports.sh file:

#!/usr/bin/env bash

# location of Saxon jar file
export CP=/home/kosarko/.m2/repository/net/sf/saxon/Saxon-HE/10.3/Saxon-HE-10.3.jar
# MD_ONLY=FALSE means only the metadata will be imported, not video files
export MD_ONLY=FALSE
export DATADIR=/home/mkuchtiak/NFA/BK_2025_MIZEJICI_REMESLA/DATA
# Handle prefix for newly imported items
export PREFIX=20.500.12801
export CONTACT_PERSON="Technical;Contact;lindat@nfa.cz;Národní filmový archiv"
# this corresponds to collection ID in transform.xslt file
export COLLECTION=MIZEJICI_REMESLA

The used following commands to export env variables.

   chmod a+x exports.sh
   source ./exports.sh

6. I've created an import directory, by running:

   cd ~/NFA/BK_2025_MIZEJICI_REMESLA
   ~/sources/lindat-repository-imports/NFA/scripts/run.sh >/tmp/run_out.log

This will create the /NFA/BK_2025_MIZEJICI_REMESLA/import directory in the format required by import script

7. Copied the import folder (with all its subfolders) to dspace-dev machine:

On dspace-dev machine (10.10.51.112) I've created new folder for MIZEJICI_REMESLA import:

  /home/mkuchtiak/DATA/imports/nfa_mizejici_remesla/

Then I copied import folder to dspace-dev machine:

  cd ~/NFA/BK_2025_MIZEJICI_REMESLA
  scp -r import mkuchtiak@10.10.51.112:/home/mkuchtiak/DATA/imports/nfa_mizejici_remesla/

8. Then I manually created a new Disappearing Traditional Crafts collection on dspace-dev.

9. then I've exported collection id on 10.10.51.112 by:

export COL_ID=123456789/6141

and finally run the import script on dspace-dev (10.10.51.112) machine:

   cd ~/../okosarko/sources/just_for_docker/lindat/
   docker compose -p lindatrepo --env-file .env -f docker/docker-compose.yml -f docker/docker-compose-rest.yml -f docker/cli.yml run -v /home/mkuchtiak/DATA/imports/nfa_mizejici_remesla/import:/import --rm dspace-cli import -a -e kuchtiak@ufal.mff.cuni.cz -c "$COL_ID" -s /import/ -m /import/mapfile_mizejici_remesla.txt

10. Detecting Duplicities:

• during the import (step 9), I found 3900417-05 item (20.500.12801/3900417-05 handle) already exists in dspace-dev repository
• I've separated data to duplicates folder:

  cd ~/DATA/imports/nfa_mizejici_remesla
  mkdir duplicates
  mv import/3900417-05 duplicates/

• repeated step 9 until it completes successfully

11. Export metadata for duplicated items

• exported metadata from dspace docker container:

  docker exec -u root -ti dspace5 bash
  cd /dspace/bin
  ./dspace metadata-export -i 20.500.12801/3900417-05 -f ../csv-exports/3900417-05.csv

• copied csv metadata export files from docker container to dspace-dev

cd /home/mkuchtiak/DATA/imports/nfa_mizejici_remesla
docker cp dspace5:/dspace/csv-exports .

12. Metadata Update

• on (ufal-point-dev) switch to scripts directory

cd ~/sources/lindat-repository-imports/NFA/scripts

• update exporter-metadata-update file, to contain new collection:

    private static String subjectMapYaml = """
        "20.500.12801/2": "MNICHOV"
        "20.500.12801/3": "GALERIE"
        "20.500.12801/4": "HEYDRICH"
        "20.500.12801/5": "VZNIK_CSR"
        "20.500.12801/6": "VYZNAMNE_POHRBY"
        "20.500.12801/7": "KURATORIUM"
        "20.500.12801/8": "NARODNI_DIVADLO"
        "20.500.12801/9": "ZDRAVOTNI_SOCIALNI"
        "20.500.12801/10": "USPORNA_OPATRENI"
        "20.500.12801/11": "STAVEBNI_RUCH"
        "123456789/6141": "MIZEJICI_REMESLA"
        "MNICHOV": "Mnichovská dohoda"
        "GALERIE": "Galerie osobností"
        "HEYDRICH": "Heydrichiáda"
        "VZNIK_CSR": "Vznik ČSR"
        "VYZNAMNE_POHRBY": "Významné pohřby"
        "KURATORIUM": "Kuratorium"
        "NARODNI_DIVADLO": "Národní divadlo"
        "ZDRAVOTNI_SOCIALNI": "Zdravotní a sociální péče"
        "USPORNA_OPATRENI": "Úsporná opatření"
        "STAVEBNI_RUCH": "Pražský stavební ruch"
        "NARODNI_PAMET": "Národní paměť"
        "KVETEN_1945": "Květen 1945"
        "MIZEJICI_REMESLA": "Mizející řemesla"
        """

• run metadata update:

cd /home/mkuchtiak/DATA/imports/nfa_mizejici_remesla
jbang ~/sources/lindat-repository-imports/NFA/scripts/exporter-metadata-update --csv-in csv-exports/3900417-05.csv --metadata-dir METADATA --csv-out 3900417-05-new.csv --map-to-collection 123456789/6141

Here

• --csv-in points to formerly exported file(s)
• --metadata-dir points to METADATA-DIR (with original metadata)
• --csv-out contains the name ow new updated metadata
• --map-to-collection contains the handle of the new "Disappearing Traditional Crafts" collection

13. Import Updated Metadata

• copy new metadata to container:

scp 3900417-05-new.csv 10.10.51.112:/home/mkuchtiak/DATA/imports/nfa_mizejici_remesla/
ssh 10.10.51.112 
cd ~DATA/imports/nfa_mizejici_remesla/
docker cp 3900417-05-new.csv dspace5:/dspace/csv-imports/

• run metadata-import on container:

  docker exec -u root -ti dspace5 bash
  cd /dspace/bin
  # validation only
  ./dspace metadata-import -f ../csv-imports/3900417-05-new.csv -v 
  # real metadata import
  ./dspace metadata-import -f ../csv-imports/3900417-05-new.csv

• finally check the metadata update on 20.500.12801/3900417-05 item in dspace-dev repository, and whether the item was mapped properly to "Disappearing Traditional Crafts" collection.