-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Configuration for training with CLAP embeddings #441
Comments
Just adding a bit more info, I managed to at least get to an attempt to load
But loading the I also tried just pointing it to the folder (it complained that it was not a file) and the |
Okay, I can load |
I worked out a way around loading the HF weights. Now what I'm wondering about is how to configure a text prompt for running test generations during training. My goal is to test the performance of training on CLAP audio embeddings and using text embeddings for inference. Any help greatly appreciated. |
In audiocraft, 'test generation' during training is little bit tricky, which is done at the following code part. audiocraft/audiocraft/solvers/musicgen.py Line 493 in 69fea8b
We may have to prepare a dataset for generation as same as training data. If you don't need to do 'continuation generation' during training, dummy audio should be enough.
|
Thanks so much for the reply! Digging around the solver code (as you pointed out) it did seem like the joint embedding might want a prompt, so I did add some super simple metadata files. I haven't run it to the point of a test generation yet, but hopefully it works as expected. I haven't added any dummy audio at this point, but I think in the past it's just used the audio from a data.... (I think...??) Another "gotcha" that wasn't obvious to me at first is that |
Actually though... what determines when it will generate a sample output? I can see it running through train and valid steps, and it's saving checkpoints, but I don't seem to be getting audio. I also want audio sent to wandb, ideally... I do have |
It seems that 'test generation' runs at end of every epoch as same as evaluation, which is defined in the BaseSolver class (base class of every solver class). audiocraft/audiocraft/solvers/base.py Line 466 in 69fea8b
As shown in the method, you can firstly check if your run go through self.should_run_stage('generate') statement. If not, it means 'test generation' is rejected here, so you can find which configuration causes the rejection. And then, finally audio saving is done in the above mentioned method generate_audio after generating audio samples at the following line: audiocraft/audiocraft/solvers/musicgen.py Line 562 in 69fea8b
|
Yes, I saw from another Issue/comment that the "every" in the "generate" config refers to epochs, not steps. I had it set to 1000, thinking it meant step, so I would have been waiting a while... heh... |
I'm wondering if anyone has any configuration info they could share on training with CLAP embeddings?
I want to try the
laion/larger_clap_music
model from Huggingface, but it's really unclear to me how the project is supposed to be configured.Any help greatly appreciated.
The text was updated successfully, but these errors were encountered: