4stems model not doing anything?

[rcgale]

Description

I'm having a great time playing with Spleeter, and things are working great with the 2stems and 5stems models, but the couple tries I've had with the 4stems model results in four files with identical contents.

Step to reproduce

Example usage I've tried:

1. Installed using conda env create -f spleeter/conda/spleeter-cpu.yaml
2. Run as spleeter separate -p spleeter:2stems -i ~/Downloads/12\ -\ Step\ into\ a\ World\ \(Rapture\'s\ Delight\).mp3 -o krs2
3. Works as expected there. However,
4. Run as spleeter separate -p spleeter:4stems -i ~/Downloads/12\ -\ Step\ into\ a\ World\ \(Rapture\'s\ Delight\).mp3 -o krs4
5. Got no error, just four identical files.

Output

# 2stems output
# This works as expected:
> diff krs2/12\ -\ Step\ into\ a\ World\ \(Rapture\'s\ Delight\)/{accompaniment,vocals}.wav
Binary files krs2/12 - Step into a World (Rapture's Delight)/accompaniment.wav and krs2/12 - Step into a World (Rapture's Delight)/vocals.wav differ

# 4stems output
# This yields no output:
> diff krs4/12\ -\ Step\ into\ a\ World\ \(Rapture\'s\ Delight\)/{drums,vocals}.wav

Full output of 4stems attempt:

> spleeter separate -p spleeter:4stems -i ~/Downloads/12\ -\ Step\ into\ a\ World\ \(Rapture\'s\ Delight\).mp3 -o krs4
INFO:tensorflow:Using config: {'_model_dir': 'pretrained_models/4stems', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': gpu_options {
  per_process_gpu_memory_fraction: 0.7
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x146d5b4a8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:tensorflow:Could not find trained model in model_dir: pretrained_models/4stems, running initialization to predict.
INFO:tensorflow:Calling model_fn.
WARNING:tensorflow:From /miniconda3/envs/spleeter-cpu/lib/python3.7/site-packages/spleeter/model/functions/unet.py:29: The name tf.keras.initializers.he_uniform is deprecated. Please use tf.compat.v1.keras.initializers.he_uniform instead.

INFO:tensorflow:Apply unet for vocals_spectrogram
INFO:tensorflow:Apply unet for drums_spectrogram
INFO:tensorflow:Apply unet for bass_spectrogram
INFO:tensorflow:Apply unet for other_spectrogram
INFO:tensorflow:Done calling model_fn.
WARNING:tensorflow:From /miniconda3/envs/spleeter-cpu/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:1354: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
WARNING:tensorflow:The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.string
WARNING:tensorflow:The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.int32
WARNING:tensorflow:The dtype of the watched tensor must be floating (e.g. tf.float32), got tf.string
INFO:tensorflow:Loading audio b"/Users/galer/Downloads/12 - Step into a World (Rapture's Delight).mp3" from 0.0 to 600.0
INFO:tensorflow:File krs4/12 - Step into a World (Rapture's Delight)/vocals.wav written
INFO:tensorflow:File krs4/12 - Step into a World (Rapture's Delight)/drums.wav written
INFO:tensorflow:File krs4/12 - Step into a World (Rapture's Delight)/bass.wav written
INFO:tensorflow:File krs4/12 - Step into a World (Rapture's Delight)/other.wav written

Environment

2019 Macbook Pro

OS	MacOS
Installation type	Conda
RAM available	several gigs
Hardware spec	CPU

Additional context

Great tool though, thanks!

[alreadytaikeune]

Hi @rcgale thanks for the feedback. After a quick look it seems something is going wrong when downloading the 4-stems model. INFO:tensorflow:Could not find trained model in model_dir: pretrained_models/4stems, running initialization to predict.. The weights of the different models get initialized to the same values hence producing equal outputs. I'll investigate why the download fails.

[rcgale]

The weights of the different models get initialized to the same values hence producing equal outputs.

Interesting! Out of curiosity, is there a rationale for this behavior? Or is this just a side effect of how it works?

Thinking through how I would want it to behave in my use cases, I would prefer moving anything right at the threshold to the "Accompaniment/Other" file, and nowhere else

As a hypothetical example, if there were a very annoying organ line in an otherwise incredible KRS-One song, I would want to have it isolated away to one track, and missing from all other tracks if possible. ;)

I haven't read up yet on how the model works though, so I understand that it might be unreasonably difficult.

[hassanamin994]

Just faced a similar issue today, where the pretrained models of 2stems/4stems dont get downloaded correctly. so downloaded them manually and added them in a directory with the name of "pretrained_models" in the same directory that i issue the spleeter command from and it worked.
i guess it's a path issue somewhere in spleeter ( where spleeter expects the pretrained_models directory to be in the same location of the command execution ).
Not totally sure of the correct fix for this as i'm not a python/conda guru ( just integrating the tool in a nodejs worker )

Awesome tool btw :)

[Faylixe]

The weights of the different models get initialized to the same values hence producing equal outputs.

Interesting! Out of curiosity, is there a rationale for this behavior? Or is this just a side effect of how it works?

Side effect.

Thinking through how I would want it to behave in my use cases, I would prefer moving anything right at the threshold to the "Accompaniment/Other" file, and nowhere else

As a hypothetical example, if there were a very annoying organ line in an otherwise incredible KRS-One song, I would want to have it isolated away to one track, and missing from all other tracks if possible. ;)

I haven't read up yet on how the model works though, so I understand that it might be unreasonably difficult.

You can totally train your model using appropriate dataset if you have one, please check the training section of the repository wiki and if you are facing some trouble during training or want to discuss about obtained results please open a dedicated issue :).