How to generate training pkls and theme files from POP909

[2000222]

Hello, your work is so brilliant and I've completed the inference process for some provided theme files (e.g., 875_theme.mid).
But I feel a little confused how to generate the similar theme file from POP909 and how to obtain the training pkl files (train_seg2_512.pkl & val_seg2_512.pkl) from scratch.
If possible, could you pls share some related ideas or scripts? Thank you~~

[atosystem]

Dear @2000222

So sorry for the late reply...

Thanks for trying out Theme Transformer!

I have to apologize for the absence of the Theme retrieval code.

For Theme midi file

Although the theme retrieval code is not available in this repo, you can still create your own Theme MIDI files by following the following format (which is seen in./theme_files/ )

(which contains 3 tracks: naming “MELODY”, “PIANO” and “Theme info track”)

For example, the following is the screenshot of ./theme_files/874_theme.mid

As for the length of the theme (number of bars), we use the length of 2 bars in our work, but the model should work without error with longer themes.

However, we do not guarantee the quality of the themes beyond POP909 dataset or themes longer than 2 bars, as our model is trained on 2-bar segment themes in POP909.

You can inference music by our model with your custom theme MIDI files.

For .pkl files

The pkl files contain REMI sequences for training and testing of our model. They are attained by gathering a bunch of remi sequences from annotated individual midi files.

For midi annotations:

Each midi file must contain 5 tracks. (The first 3 tracks directly come from POP909 only preprocessed with 1/16 bar quantize)

others:

• boundary_track : indicates the boundary of each music segment. (We employ 2-bar segmentation in our paper. You are welcome to use other phrase segmentation methods). The onset and offset of the notes correspond to the start and the end of the music segment. The pitch has no physical meaning, but please use the same value for every segment.

• melody_boundary_track (Optional, you can leave it blank): indicates the cluster for each music segment. The segments with same pitch are in the same cluster (they sound similar). Notice that the lowest pitch for the notes in this track must be the pitch in boundary_track

• theme info track : indicates the music segments that are selected as theme

Because we do not provide the theme retrieval code currently, you must decide the theme region by yourself, as well as boundaries.

boundary_track	melody_boundary_track	theme info track

After the annotations are done, please call the following 2 functions

• midi2REMI : converting midifile to REMI sequence
  

  ThemeTransformer/preprocess/vocab.py

  Lines 158 to 173 in 5ad990d

  	def midi2REMI(self,midi_path,quantize=True,trim_intro = True,trim_outro=True,include_bridge=False,extend_theme=False,bar_first=False,theme_annotations=True,verbose = False):
  	"""convert midi file to token representation
  	Args:
  	midi_path (str): the path of input midi file
  	trim_intro (bool, optional): ignore the intro part of the song. Defaults to True.
  	trim_outro (bool, optional): ignore the outro part of the song. Defaults to True.
  	include_bridge (bool, optional): ignore the intro part of the song. Defaults to False.
  	extend_theme (bool, optional): extend the theme region to at least MIN_MEL_NOTES=8 notes. Defaults to False.
  	bar_first (bool, optional): place Bar token in front of Theme-Start token. Defaults to False.
  	theme_annotations (bool, optional): including theme-realted tokens. Defaults to True.
  	verbose (bool, optional): print some message. Defaults to False.
  	Returns:
  	list: sequence of tokens
  	"""

• preprocessREMI : preprocess REMI sequence for model input
  

  ThemeTransformer/preprocess/vocab.py

  Lines 360 to 378 in 5ad990d

  	def preprocessREMI(self,remi_sequence,always_include=False,max_seq_len=512,strict=True,verbose=False):
  	"""Preprocess token sequence
  	slicing the sequence for training our models
  	Args:
  	remi_sequence (List): the music token seqeunce
  	always_include (Bool): selected the data including either Theme-Start or Theme-End
  	max_seq_len (Int): maximum sequence length for each data
  	strict (Bool): the returning sequence should always include Theme-Start
  	Return:
  	{
  	"src" : <corressponding theme condition>,
  	"src_theme_binary_msk" : <corressponding theme condition's theme msk>,
  	"tgt_segments" : <list of target sequences>,
  	"tgt_segments_theme_binary_msk" : <list of target sequences theme msk>
  	}
  	"""

Example:
For each midi:

remi_seq = myvocab.midi2REMI(midifp,include_bridge=False,bar_first=False,verbose=False)
ret = myvocab.preprocessREMI(remi_seq,always_include=True,max_seq_len=512,verbose=True)

Finally, you can collect all ret for each song and insert them into a list and save it to pkl.

Again, Thanks a lot for using Theme Transformer~

[2000222]

Thanks a lot for your tutorial. 💛 Exactly, I've created a manual theme_file with three tracks inspired by my favorite singer and applied the ThemeTransformer to generate new music.
😄 Hope a good publication and presentation for your brilliant work ~