examples of creation.sh, import.py and metadata.py

[alecristia]

Here are a few examples of bash scripts documenting the creation of datasets:

Example 1: EL1000 dataset with annotations that needed to be anonymized

#steps taken to create this

# it was not my first time using childproject, but if it had been, I would have followed: https://childproject.readthedocs.io/en/latest/install.html

# for stand-alone repos, I would have:
# datalad create rague
# but since this one follows the EL1000 template, I looked at:
# https://gin.g-node.org/EL1000/EL1000/issues/3
# and followed https://github.com/LAAC-LSCP/datalad-procedures/blob/alecristia-patch-1/README.md#the-el1000-template


# if needed
# source ~/ChildProjectVenv/bin/activate
# cd rague

# NOTE! this is the alternative for annotations that need to be anonymized 

#### SECTION ONE: CREATING LOCAL COPIES OF FILES #### 
# -- .its are put in confidential/raw/ because they need to be used for 
# creating metadata, but note that later they'll be moved elsewhere
mkdir -p annotations/its/confidential/raw/
cp /Users/acristia/Box/data_broad_strokes_lena/rague/ITS_FILES_OSF_3.15.2019/*.its annotations/its/confidential/raw/.
mkdir -p metadata/confidential/original/
cp /Users/acristia/Box/data_broad_strokes_lena/rague/purdue_ndd_metadata.csv metadata/confidential/original/.
# I put all the other files not touched above in extra
cp /Users/acristia/Box/data_broad_strokes_lena/rague/purdue_ndd_readme.txt extra/.


#### SECTION TWO: CREATE METADATA #### 
# working on the metadata
# I took metadata.py from lyon (within EL1000)
python scripts/metadata.py rague metadata/confidential/original/purdue_ndd_metadata.csv --recompute-ages

child-project validate . --ignore-files

datalad save . -m "first commit"

#### SECTION THREE: IMPORT ITS #### 

#now we go back to annotations, to anonymize
child-project anonymize . --input-set its/confidential --output-set its

#next import its
# I took import.py from lyon (within EL1000)
python scripts/import.py its

# we don't do:
# child-project compute-durations
# because we don't have the raw files

child-project validate . --ignore-files --check-annotations


#### SECTION FOUR: WRAP UP #### 

datalad save . -m "validated"
datalad push

Example 2: EL1000, no need to anonymize

# it was not my first time using childproject, but if it had been, I would have followed: https://childproject.readthedocs.io/en/latest/install.html

# for stand-alone repos, I would have:
# datalad create lyon
# but since this one follows the EL1000 template, I looked at:
# https://gin.g-node.org/EL1000/EL1000/issues/3
# and followed https://github.com/LAAC-LSCP/datalad-procedures/blob/alecristia-patch-1/README.md#the-el1000-template


# if needed
# source ~/ChildProjectVenv/bin/activate
# cd lyon

# NOTE! this is the alternative for annotations that DO NOT need to be anonymized 
mkdir -p annotations/its/raw/
cp /Users/valentinthouzeau/Documents/projets/SES_language/data_EL1000/lyon/*/*.its annotations/its/raw/.

# working on the metadata
# I took metadata.py from bergelson (within EL1000)
mkdir -p metadata/original/
cp /Users/valentinthouzeau/Documents/projets/SES_language/data_EL1000/lyon/lyon_metadata.csv metadata/original/.
python scripts/metadata.py lyon metadata/original


#next import its
# I took import.py from bergelson (within EL1000)
python scripts/import.py its



# I put all the other files not touched above in extra
cp /Users/acristia/Box/data_broad_strokes_lena/rague/purdue_ndd_readme.txt extra/.


child-project validate . --ignore-files

datalad save . -m "first commit"

# we don't do:
# child-project compute-durations
# because we don't have the raw files


child-project validate . --ignore-files --check-annotations

datalad save . -m "checked"

touch recordings/.gitignore 

datalad save recordings -m "recordings directory"

example import.py

#!/usr/bin/env python3

import argparse
import os
import pandas as pd
from EL1000.annotations import AnnotationImporter

def get_filter(row):
    its = row['its_filename']

    if row['set'] == 'vtc':
        table = pd.read_csv('metadata/original/lyon_dict.csv')
        table = table[table['its'] == its]
        f = table['hb_cha_name'].iloc[0]
        return f
    else:
        return None

def get_raw_name(row):
    if row['set'] == 'vtc':
        return 'all.rttm'
    elif 'cha' in row['set']:
        its = row['its_filename']
        table = pd.read_csv('metadata/original/lyon_dict.csv')
        table = table[table['its'] == its]
        f = table['hb_cha_name'].iloc[0] + '.cha'
        return f
    else:
        return row['raw_filename']

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description = 'import annotations')
    parser.add_argument('set', help = 'set', choices = ['its', 'vtc', 'cha/an1'])
    parser.add_argument('--threads', help = 'how many threads', default = 4, type = int)
    args = parser.parse_args()

    filter = None

    if args.set != 'its':
        filter = get_filter

    importer = AnnotationImporter('.')
    importer.process(
        set = args.set,
        threads = args.threads,
        filter_func = filter,
        raw_name_func = get_raw_name,
        format = 'cha' if 'cha' in args.set else None
    )

Example metadata.py

#!/usr/bin/env python3

import pandas as pd
from EL1000.metadata import MetadataImporter

ses_table = {
    'bacc +3': '3',
    'bacc +4': '4',
    'bacc +5': '5',
    'PHD': '5'
}

if __name__ == "__main__":
    importer = MetadataImporter(
        experiment = 'lyon',
        metadata = 'metadata/original/lyon_metadata.csv',
        preserve_dates = True,
        preserve_metadata = True
    )

    its = pd.read_csv(importer.metadata, dtype = {'child_id': str}, encoding = 'Latin-1')
    its.columns = list(map(lambda s: s.lower(), its.columns))
    its.rename(columns = {
        'its': 'its_filename',
        'sex': 'child_sex',
        'language': 'languages'
    }, inplace = True)

    its['mat_ed'] = its['ses']
    its['original_ses'] = its['ses']
    its['ses'] = its['ses'].astype(str).map(ses_table)

    its['languages'] = its['languages'].apply(lambda s: ','.join(s.split('_')))

    children, recordings = importer.process(its)

    children.drop(columns = children.columns & {'its_filename', 'date_iso', 'age', 'hb_cha_name'}, inplace = True)
    children.to_csv('metadata/children.csv')

    recordings.set_index('recording_filename', inplace = True)
    recordings.to_csv('metadata/recordings.csv')