adding recordings to a dataset & updating it #270
Replies: 14 comments 60 replies
-
This is right
|
Beta Was this translation helpful? Give feedback.
-
Question: how problematic could it be if I calculated eg recording duration and therefore onset time of recordings other than the first one with formulas in excel? It may be closest to what other people will do with their own data |
Beta Was this translation helpful? Give feedback.
-
To update recordings.csvm I did the following
|
Beta Was this translation helpful? Give feedback.
-
my next step is to convert recs. I'll do this in the cluster, with this command:
Some things of note:
I haven't launched it because I'm afraid this will overwrite the other files that have already been converted AND I'm not sure whether I need to "get" all the raw files (rather than just the ones I want to convert. Perhaps I should still get them all, so that I can do a full test of the dataset? |
Beta Was this translation helpful? Give feedback.
-
Not sure why, I'll investigate
… Le 9 sept. 2021 à 19:00, Alex Cristia ***@***.***> a écrit :
one more: I did child-project compute-durations ., which I thought would compute durations and put them in recordings.csv in the right format, but I'm still getting things like:
warning: ./metadata/recordings.csv: '4689.66' does not match the format required for 'duration' on line 198, expected '([0-9]+)'
Oh and I should add that I think that the tool DID calculate duration because line 198 is one where I had entered an integer by hand, and now it has a decimal.
I'm making sure all is well then pushing my version so you can look at it on GIN.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
Beta Was this translation helpful? Give feedback.
-
next step is to analyze the data from the new recs using VTC. In my home, I did
But I'm not sure of the next step. It could be or rather run_vtc.py --inputs (files I want to analyze) --destinations (...?) Thanks in advance for the help! |
Beta Was this translation helpful? Give feedback.
-
The next step is importing the vtc output. There is already a script for this in
It doesn't seem to require any arguments, so I decide to give it a go. Before that, Then I try Since it doesn't take up much space, for simplicity I just get all the other rttms I was missing:
I could probably have created a version of the So I try
Just in case, before proceeding, I do I get the same 180 errors from before, which are due to my not having the recs locally. So all is good, |
Beta Was this translation helpful? Give feedback.
-
I'm skipping ALICE, so I can already update the metrics that are estimated. To this end, I opened the README in the
... which is fixed by There is no ALICE or VCM annotation in this project so it is normal to get this warning:
I opened the vanuatu.csv created and it has my new 3 kids -- so success! |
Beta Was this translation helpful? Give feedback.
-
The next step is to set up a zooniverse annotation process. After reading the tutorial the best seems to create a dataset where I adapted steps 1 and 3 of #251 & the tutorial:
I then create a record of how I selected in a file launched with
This is taking a while, so I'll report back on it tomorrow! (I didn't want to take up a lot of space on the server, so I went into my other version of the |
Beta Was this translation helpful? Give feedback.
-
by now I've gotten all the happy messages of extraction, so the next step is to extract chunks, which I did with
This selection may annoy coders, because the majority of that will be silent, so I'm editing the instructions to apologize for it in advance. NOTE that 40 windows x 40 children x 30 seconds x 2 chunks/sec leads to 96k chunks to be uploaded to zooniverse. This requires extra permissions, which we have obtained in the past. Nonetheless, I'll upload 1k to get started and make sure all is well. The next step is uploading. I do the first two steps in the command line (export ZOONIVERSE_LOGIN=""
at this point I hit an error:
To trouble-shoot, I wanted to share my status on GIN. Before doing so, I made sure to create a .gitignore, with the following contents:
This allowed me to not share the mp3/wav, which would have taken forever (and would have been redundant, since these are extracts from the long files). The extracts look like:
So they look fine. I also want to make sure -- when they are pushed onto Zooniverse, they'll be giving a new name, right? I thought must have done something wrong because I ended up with 2 layers of "chunks" folders: my actual mp3s are inside However, there is no csv in So this suggests something in the previous step went wrong. I tried
So then I
I wonder if the problem isn't that I'm being too ambitious. I decide to go back to square one:
(selection in progress -- to be updated tomorrow...) |
Beta Was this translation helpful? Give feedback.
-
One more hint: since you are using slurm, you could use the --threads
option of extract-chunks (e.g. --threads 32) and use as many CPUs.
(--cpus-per-task=32
or -c 32)
Recordings will be processed in parallel.
Le jeu. 16 sept. 2021 à 18:13, Alex Cristia ***@***.***> a
écrit :
… great tip, trying it out now. To this end, my extract file now looks like
this:
#!/bin/sh
child-project zooniverse extract-chunks vanuatu \
--keyword loud \
--chunks-length 500 \
--segments samples/loud_5s_10perchild/segments_20210916_151531.csv \
--destination samples/loud_5s_10perchild/chunks \
--profile standard
And I called it like this:
sbatch --mem=64G --time=5:00:00 --cpus-per-task=4 --ntasks=1 -o
extract.txt ./scripts/extract.sh
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#270 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABI4QHQ4PGYAHBXKFXANSRDUCIJSHANCNFSM5DBJLFMA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Lucas Gautheron
06 79 23 86 47
Laboratoire de Sciences Cognitives et Psycholinguistique
Bureau 414 - 29 rue d'Ulm
75005 Paris
|
Beta Was this translation helpful? Give feedback.
-
Separately, I thought I could also very easily re-run reliability because there was already a reliability.py in
I got an error in reliab.txt:
I think this is because I should activate some additional environment? to be continued too! |
Beta Was this translation helpful? Give feedback.
-
Interesting:) Well this really has to be the fact that the file is empty.
I'll try to figure out why it is empty.
Le lun. 20 sept. 2021 à 19:16, Alex Cristia ***@***.***> a
écrit :
… good question! it's zero bytes. Perhaps we're trying to look beyond the
duration of the audio file?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#270 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABI4QHQIIEKFN37FBPDTE4LUC5UAVANCNFSM5DBJLFMA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Lucas Gautheron
06 79 23 86 47
Laboratoire de Sciences Cognitives et Psycholinguistique
Bureau 414 - 29 rue d'Ulm
75005 Paris
|
Beta Was this translation helpful? Give feedback.
-
Explanation:
If the output file is not locally present, the repository only contains a
dangling pointer.
This pointer is not writable.
So you've got two options:
1) use datalad unlock to replace the pointer with the contents of the
file, but that means you need to `datalad get` the file first.
2) or use rm extra/reliability.png to remove the pointer first.
Le lun. 20 sept. 2021 à 19:28, Alex Cristia ***@***.***> a
écrit :
… I got a new and odd error:
/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexing.py:1743:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
isetter(ilocs[0], value)
/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexing.py:1596:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
self.obj[key] = _infer_fill_value(value)
Traceback (most recent call last):
File "/var/spool/slurmd/job223956/slurm_script", line 58, in
plt.savefig('extra/reliability.png', bbox_inches = 'tight')
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/pyplot.py",
line 966, in savefig
res = fig.savefig(*args, **kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/figure.py",
line 3015, in savefig
self.canvas.print_figure(fname, **kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/backend_bases.py",
line 2261, in print_figure
**kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/backend_bases.py",
line 1669, in wrapper
return func(*args, **kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py",
line 511, in print_png
dpi=self.figure.dpi, metadata=metadata, pil_kwargs=pil_kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/image.py",
line 1616, in imsave
image.save(fname, **pil_kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/PIL/Image.py",
line 2232, in save
fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory:
'extra/reliability.png'
Odd that an output file is required -- just in case, I datalad get
extra/reliability.png and launched the process again.
I then got
PermissionError: [Errno 13] Permission denied: 'extra/reliability.png'
so I did datalad unlock extra/reliability.png and this time I succeeded:
/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexing.py:1743:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
isetter(ilocs[0], value)
/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexing.py:1596:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
self.obj[key] = _infer_fill_value(value)
I also added a line to the script in order to write out the confusion
matrix as a csv!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#270 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABI4QHX5FOZB4CTZU7OPF4DUC5VNFANCNFSM5DBJLFMA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Lucas Gautheron
06 79 23 86 47
Laboratoire de Sciences Cognitives et Psycholinguistique
Bureau 414 - 29 rue d'Ulm
75005 Paris
|
Beta Was this translation helpful? Give feedback.
-
Vanuatu now has data for 3 more children than before.
The children are already listed in children, and I added the raw recordings.
The order of operations now is:
While at it, I thought I could also:
Beta Was this translation helpful? Give feedback.
All reactions