adding recordings to a dataset & updating it #270
Replies: 14 comments 60 replies
-
This is right
|
Beta Was this translation helpful? Give feedback.
-
|
Question: how problematic could it be if I calculated eg recording duration and therefore onset time of recordings other than the first one with formulas in excel? It may be closest to what other people will do with their own data |
Beta Was this translation helpful? Give feedback.
-
|
It is what I had to do for the bulk of the recordings; however, you need to flag |
Beta Was this translation helpful? Give feedback.
-
|
To update recordings.csvm I did the following
|
Beta Was this translation helpful? Give feedback.
-
|
This all sounds good, except maybe that |
Beta Was this translation helpful? Give feedback.
-
|
(Also, |
Beta Was this translation helpful? Give feedback.
-
|
ah I'll definitely do that because although I expressed them in ms, ffmpeg rounds to centiseconds, so they are not perfect. But just to clarify: I get the impression that |
Beta Was this translation helpful? Give feedback.
-
|
for consistency, everything should be expressed in milliseconds. I checked again, and this seems to be the case for previous recordings, so updating durations for the 3 children you added should be enough! |
Beta Was this translation helpful? Give feedback.
-
|
my next step is to convert recs. I'll do this in the cluster, with this command:
Some things of note:
I haven't launched it because I'm afraid this will overwrite the other files that have already been converted AND I'm not sure whether I need to "get" all the raw files (rather than just the ones I want to convert. Perhaps I should still get them all, so that I can do a full test of the dataset? |
Beta Was this translation helpful? Give feedback.
-
|
I tried the command with recordings and without the split argument
And I get a different error for with --skip-existing (and still without the split argument), which can be fixed by doing
And if you forget to Whereas if you do do datalad get + unlock + --skip-existing without the --split switch, you get a happy message, in my case written into vanuatu.txt:
However, at least in my case, The error suggests file not found: `recordings/raw/child9/REC001.WAV : No such file or directory`` I checked whether there is anything weird with that file:
And whether it's maybe not really there:
I don't think you need to unlock files to be able to read them, but I tried that:
And then I re-launched my batch, which was done instantly, created a yaml file, and Not sure how to proceed... |
Beta Was this translation helpful? Give feedback.
-
|
there are extra spaces at the end of this error can be spotted by doing EDIT: this kind of error is hard to spot when using a spread-sheet editor, but much easier to find with a text editor. |
Beta Was this translation helpful? Give feedback.
-
|
quick question: would you recommend getting ALL the files in my case, so I can fully trust the output of
(I also wonder whether the check for local files looks only at raw, which makes sense) Note that conceptually at least, I don't want to avoid checking for files altogether, since I do have some local files. Perhaps instead of an error file missing, we could get a warning -- file is a link? |
Beta Was this translation helpful? Give feedback.
-
|
also, any idea why I may be getting errors like: |
Beta Was this translation helpful? Give feedback.
-
|
one more: I did
Oh and I should add that I think that the tool DID calculate duration because line 198 is one where I had entered an integer by hand, and now it has a decimal. I'm making sure all is well then pushing my version so you can look at it on GIN. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure why, I'll investigate
… Le 9 sept. 2021 à 19:00, Alex Cristia ***@***.***> a écrit :
one more: I did child-project compute-durations ., which I thought would compute durations and put them in recordings.csv in the right format, but I'm still getting things like:
warning: ./metadata/recordings.csv: '4689.66' does not match the format required for 'duration' on line 198, expected '([0-9]+)'
Oh and I should add that I think that the tool DID calculate duration because line 198 is one where I had entered an integer by hand, and now it has a decimal.
I'm making sure all is well then pushing my version so you can look at it on GIN.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
Beta Was this translation helpful? Give feedback.
-
|
Ok, here are the reasons:
|
Beta Was this translation helpful? Give feedback.
-
humans... |
Beta Was this translation helpful? Give feedback.
-
|
next step is to analyze the data from the new recs using VTC. In my home, I did But I'm not sure of the next step. It could be or rather run_vtc.py --inputs (files I want to analyze) --destinations (...?) Thanks in advance for the help! |
Beta Was this translation helpful? Give feedback.
-
|
Oh, I get it! again, my bad -- the script stops in case of errors. Let me change that |
Beta Was this translation helpful? Give feedback.
-
|
Done. You can do |
Beta Was this translation helpful? Give feedback.
-
|
full output, outside of conda:
[7 rows x 15 columns] [7 rows x 21 columns] |
Beta Was this translation helpful? Give feedback.
-
|
Seems good! I just killed my own jobs, to make some room for you |
Beta Was this translation helpful? Give feedback.
-
|
OK looks like that worked because I have normal-looking rttms inside
which should be treated as normal The same for the contents of
To similarly analyze the other new recs, I did: Then I get the "No module named 'ChildProject'" -- which we had seen before. So the above is NOT the right order of operations. I do And now it runs:
|
Beta Was this translation helpful? Give feedback.
-
|
The next step is importing the vtc output. There is already a script for this in It doesn't seem to require any arguments, so I decide to give it a go. Before that, Then I try Since it doesn't take up much space, for simplicity I just get all the other rttms I was missing:
I could probably have created a version of the So I try
Just in case, before proceeding, I do I get the same 180 errors from before, which are due to my not having the recs locally. So all is good, |
Beta Was this translation helpful? Give feedback.
-
|
I'm skipping ALICE, so I can already update the metrics that are estimated. To this end, I opened the README in the
... which is fixed by There is no ALICE or VCM annotation in this project so it is normal to get this warning:
I opened the vanuatu.csv created and it has my new 3 kids -- so success! |
Beta Was this translation helpful? Give feedback.
-
|
The next step is to set up a zooniverse annotation process. After reading the tutorial the best seems to create a dataset where I adapted steps 1 and 3 of #251 & the tutorial: I then create a record of how I selected in a file launched with
This is taking a while, so I'll report back on it tomorrow! (I didn't want to take up a lot of space on the server, so I went into my other version of the |
Beta Was this translation helpful? Give feedback.
-
|
new error, fixed similarly:
After that, I get happy messages like:
And less happy ones, like:
I should have validated the corpus before launching this! Now I do and I get 197 errors -- some are unindexed files, which I know about and it doesn't worry us. The missing file errors probably refer to the raw recordings, not the converted. I wondered whether we should think about other In any case, I did |
Beta Was this translation helpful? Give feedback.
-
|
by now I've gotten all the happy messages of extraction, so the next step is to extract chunks, which I did with This selection may annoy coders, because the majority of that will be silent, so I'm editing the instructions to apologize for it in advance. NOTE that 40 windows x 40 children x 30 seconds x 2 chunks/sec leads to 96k chunks to be uploaded to zooniverse. This requires extra permissions, which we have obtained in the past. Nonetheless, I'll upload 1k to get started and make sure all is well. The next step is uploading. I do the first two steps in the command line (export ZOONIVERSE_LOGIN="" at this point I hit an error:
To trouble-shoot, I wanted to share my status on GIN. Before doing so, I made sure to create a .gitignore, with the following contents: This allowed me to not share the mp3/wav, which would have taken forever (and would have been redundant, since these are extracts from the long files). The extracts look like: So they look fine. I also want to make sure -- when they are pushed onto Zooniverse, they'll be giving a new name, right? I thought must have done something wrong because I ended up with 2 layers of "chunks" folders: my actual mp3s are inside However, there is no csv in So this suggests something in the previous step went wrong. I tried
So then I
I wonder if the problem isn't that I'm being too ambitious. I decide to go back to square one:
(selection in progress -- to be updated tomorrow...) |
Beta Was this translation helpful? Give feedback.
-
|
Hello,
My advice is to run extract-chunks 1) with slurm or 2) within a screen (make sure to detach the screen before logging off) |
Beta Was this translation helpful? Give feedback.
-
|
great tip, trying it out now. To this end, my And I called it like this: |
Beta Was this translation helpful? Give feedback.
-
|
One more hint: since you are using slurm, you could use the --threads
option of extract-chunks (e.g. --threads 32) and use as many CPUs.
(--cpus-per-task=32
or -c 32)
Recordings will be processed in parallel.
Le jeu. 16 sept. 2021 à 18:13, Alex Cristia ***@***.***> a
écrit :
… great tip, trying it out now. To this end, my extract file now looks like
this:
#!/bin/sh
child-project zooniverse extract-chunks vanuatu \
--keyword loud \
--chunks-length 500 \
--segments samples/loud_5s_10perchild/segments_20210916_151531.csv \
--destination samples/loud_5s_10perchild/chunks \
--profile standard
And I called it like this:
sbatch --mem=64G --time=5:00:00 --cpus-per-task=4 --ntasks=1 -o
extract.txt ./scripts/extract.sh
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#270 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABI4QHQ4PGYAHBXKFXANSRDUCIJSHANCNFSM5DBJLFMA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Lucas Gautheron
06 79 23 86 47
Laboratoire de Sciences Cognitives et Psycholinguistique
Bureau 414 - 29 rue d'Ulm
75005 Paris
|
Beta Was this translation helpful? Give feedback.
-
|
that worked! Next step is upload, but before then:
It worked for a few, then I hit another error:
Of course this is due because I saved, which locked the file!! Hmm now what should I do? Because I can see on https://www.zooniverse.org/lab/14957/subject-sets/98069 that the set has been uploaded (although it says 0 subjects AND the counts do not reflect this ["The project has 127 uploaded subjects. You have uploaded 138 subjects from an allowance of 10000."], there is a bunch of them, and I can hear them). I wouldn't want these to "count" towards my allowance, but since I couldn't write to chunks, there is no local record that I uploaded them, which will cause trouble later on when I want to upload more (bc I won't know which I've already uploaded) AND I'm not sure I have a record of the name they were attributed upon upload. I would love to remove this set from zooniverse -- but you cannot do that (I'd have to remove one by one...) -- but I tagged it as REMOVE- and I won't put it into the workflow https://www.zooniverse.org/lab/14957/workflows/17576. OK! so now I unlock the chunks... csv file, rerun upload, and when that is done, datalad save to keep record of the uploads. AH note, I still get an error:
to be continued... |
Beta Was this translation helpful? Give feedback.
-
|
Can you please upload the chunks metadata files here? |
Beta Was this translation helpful? Give feedback.
-
|
Yes! Thank you. Or link to upload: |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. I posted about this issue on Zooniverse Talk. |
Beta Was this translation helpful? Give feedback.
-
|
good question! it's zero bytes. Perhaps we're trying to look beyond the duration of the audio file? |
Beta Was this translation helpful? Give feedback.
-
|
Separately, I thought I could also very easily re-run reliability because there was already a reliability.py in
I got an error in reliab.txt:
I think this is because I should activate some additional environment? to be continued too! |
Beta Was this translation helpful? Give feedback.
-
|
replicated without slurm, so now doing |
Beta Was this translation helpful? Give feedback.
-
|
after force reinstall I was still getting the error, so I just did: |
Beta Was this translation helpful? Give feedback.
-
|
I got a new and odd error:
Odd that an output file is required -- just in case, I I then got
so I did
I also added a line to the script in order to write out the confusion matrix as a csv! |
Beta Was this translation helpful? Give feedback.
-
|
Interesting:) Well this really has to be the fact that the file is empty.
I'll try to figure out why it is empty.
Le lun. 20 sept. 2021 à 19:16, Alex Cristia ***@***.***> a
écrit :
… good question! it's zero bytes. Perhaps we're trying to look beyond the
duration of the audio file?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#270 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABI4QHQIIEKFN37FBPDTE4LUC5UAVANCNFSM5DBJLFMA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Lucas Gautheron
06 79 23 86 47
Laboratoire de Sciences Cognitives et Psycholinguistique
Bureau 414 - 29 rue d'Ulm
75005 Paris
|
Beta Was this translation helpful? Give feedback.
-
|
Can you please do |
Beta Was this translation helpful? Give feedback.
-
|
there are too many files, but the key infos are:
Thanks for your help with this mystery!! |
Beta Was this translation helpful? Give feedback.
-
|
Apparently some mp3 have two channels and some others have only one. This is because the recordings, even those that were converted, seem to have different numbers of channels. Apparently those that have been concatenated underwent a process that made them mono. I suspect this should not make any difference. The stereo clips have two identical channels (just checked), so mono or stereo should play the same with a headset. The reason why a few dozens of clips failed to extract it still mysterious to me. I had zero failure on oberon. I suggest to install the latest version of the package and run extract-chunks again. It will overwrite the empty chunks. How did you run it? if you do parallel processing with slurm, make sure to use enough memory (I suggest at least 1G, maybe 2G per cpu). If it's really important to make all the recordings mono, child-project has a pipeline to do that! |
Beta Was this translation helpful? Give feedback.
-
|
oops looks like I forgot to send the message I wrote this morning. The key ideas:
In any case, why some zeros? no idea... I did: and now things look ok:
So I changed the file in I decided to do only 100 this time, to see how it goes. It worked -- I can see and hear the subjects on zooniverse, and it looks like my N of uploads has gone up by 100. I changed the name of the set and launched again, and another set appeared. I also checked and the chunks csv file has been updated with references to those uploads. So it seems like it was a freak error, but all is well! With this I nearly end my journey of updating this dataset. The only thing left is to wait until more annotations come by, and incorporate these into the dataset. That will take a while, so to be continued! |
Beta Was this translation helpful? Give feedback.
-
I do not expect stereo to be 2*mono in terms of bytes at least because of the header (which size is usually constant or almost constant). Just to make it clear, I didn't figure out the mono/stereo thing based off the sizes; I used soxi to print the metadata for all audio files, which includes the amount of channels.
All USB files were probably identical, but for some recordings I only had the raw files (REC00x.wav) and for some recordings I only had access to already processed, concatenated files (named vanuatu_etc.wav).
|
Beta Was this translation helpful? Give feedback.
-
|
Explanation:
If the output file is not locally present, the repository only contains a
dangling pointer.
This pointer is not writable.
So you've got two options:
1) use datalad unlock to replace the pointer with the contents of the
file, but that means you need to `datalad get` the file first.
2) or use rm extra/reliability.png to remove the pointer first.
Le lun. 20 sept. 2021 à 19:28, Alex Cristia ***@***.***> a
écrit :
… I got a new and odd error:
/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexing.py:1743:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
isetter(ilocs[0], value)
/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexing.py:1596:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
self.obj[key] = _infer_fill_value(value)
Traceback (most recent call last):
File "/var/spool/slurmd/job223956/slurm_script", line 58, in
plt.savefig('extra/reliability.png', bbox_inches = 'tight')
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/pyplot.py",
line 966, in savefig
res = fig.savefig(*args, **kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/figure.py",
line 3015, in savefig
self.canvas.print_figure(fname, **kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/backend_bases.py",
line 2261, in print_figure
**kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/backend_bases.py",
line 1669, in wrapper
return func(*args, **kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/backends/backend_agg.py",
line 511, in print_png
dpi=self.figure.dpi, metadata=metadata, pil_kwargs=pil_kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/matplotlib/image.py",
line 1616, in imsave
image.save(fname, **pil_kwargs)
File
"/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/PIL/Image.py",
line 2232, in save
fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory:
'extra/reliability.png'
Odd that an output file is required -- just in case, I datalad get
extra/reliability.png and launched the process again.
I then got
PermissionError: [Errno 13] Permission denied: 'extra/reliability.png'
so I did datalad unlock extra/reliability.png and this time I succeeded:
/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexing.py:1743:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
isetter(ilocs[0], value)
/scratch1/home/acristia/ChildProjectVenv/lib/python3.7/site-packages/pandas/core/indexing.py:1596:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
self.obj[key] = _infer_fill_value(value)
I also added a line to the script in order to write out the confusion
matrix as a csv!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#270 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABI4QHX5FOZB4CTZU7OPF4DUC5VNFANCNFSM5DBJLFMA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
--
Lucas Gautheron
06 79 23 86 47
Laboratoire de Sciences Cognitives et Psycholinguistique
Bureau 414 - 29 rue d'Ulm
75005 Paris
|
Beta Was this translation helpful? Give feedback.
-
Vanuatu now has data for 3 more children than before.
The children are already listed in children, and I added the raw recordings.
The order of operations now is:
While at it, I thought I could also:
Beta Was this translation helpful? Give feedback.
All reactions