Tensorsize doesn't match, svc_hubert_soft_diff_svc.py

[AWAS666]

I'm crashing on "svc_hubert_soft_diff_svc.py" with the following traceback, isn't that something similar to #86 ?

| Name | Type | Params

0 | model | DiffSinger | 32.0 M
1 | vocoder | NsfHifiGAN | 14.2 M

32.0 M Trainable params
14.2 M Non-trainable params
46.2 M Total params
184.927 Total estimated model params size (MB)
Sanity Checking DataLoader 0: 0%| | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "C:\Users\User\Documents\Testing\fishdiffusion\tools\diffusion\train.py", line 98, in
trainer.fit(model, train_loader, valid_loader, ckpt_path=args.resume)
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 520, in fit
call._call_and_handle_interrupt(
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\trainer\call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 559, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 935, in _run
results = self._run_stage()
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 976, in _run_stage
self._run_sanity_check()
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1005, in _run_sanity_check
val_loop.run()
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\loops\utilities.py", line 177, in _decorator
return loop_run(self, *args, **kwargs)
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\loops\evaluation_loop.py", line 115, in run
self._evaluation_step(batch, batch_idx, dataloader_idx)
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\loops\evaluation_loop.py", line 375, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_kwargs.values())
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\trainer\call.py", line 288, in _call_strategy_hook
output = fn(*args, **kwargs)
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 378, in validation_step
return self.model.validation_step(*args, **kwargs)
File "C:\Users\User\Documents\Testing\fishdiffusion\fish_diffusion\archs\diffsinger\diffsinger.py", line 276, in validation_step
return self._step(batch, batch_idx, mode="valid")
File "C:\Users\User\Documents\Testing\fishdiffusion\fish_diffusion\archs\diffsinger\diffsinger.py", line 191, in _step
output = self.model(
File "C:\Users\User\Documents\Testing\fishdiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\Users\User\Documents\Testing\fishdiffusion\fish_diffusion\archs\diffsinger\diffsinger.py", line 134, in forward
features = self.forward_features(
File "C:\Users\User\Documents\Testing\fishdiffusion\fish_diffusion\archs\diffsinger\diffsinger.py", line 96, in forward_features
features += self.pitch_encoder(pitches)
RuntimeError: The size of tensor a (4) must match the size of tensor b (608) at non-singleton dimension 1

[leng-yue]

It seems like your features and pitches have different shapes. Can you print their shape before this line:

features += self.pitch_encoder(pitches)

[AWAS666]

I hope this is what you've meant:
print(features.shape)
print(pitches.shape)

this prints:

torch.Size([4, 608, 256])
torch.Size([4, 608, 1])

[leng-yue]

The shape of features and pitches are correct.
Can you also print the shape of "self.pitch_encoder(pitches)" and submit your config file.

[AWAS666]

this:
print(features.shape)
print(pitches.shape)
print(self.pitch_encoder(pitches).shape)

prints:
torch.Size([4, 608, 256])
torch.Size([4, 608, 1])
torch.Size([4, 608, 1, 256])

the config is the default "svc_hubert_soft_diff_svc.py".
I mostly only changed the batch_size to 105 and num_workers to 10

[leng-yue]

Can you try configs/svc_content_vec.py instead?
I double-checked the config of svc_hubert_soft_diff_svc, and there is some issue in the pitch_to_coarse (already deprecated).

[AWAS666]

that one works just fine.
Isn't "svc_hubert_soft_diff_svc" supposed to be a better option or am I understanding something wrong here?

[leng-yue]

svc_hubert_soft_diff_svc is a very old deprecated version.

[AWAS666]

btw could you please create a comparison table in the Readme?
else it is really hard to know what is usable, what is older, what is more lightweight and what is more high quality.

[leng-yue]

Generally, newer models are better, but I think making a comparison table will be helpful. Will do it later.

[leng-yue]

We plan to release a comparison and other information in the next major release.