MELOD4J package for learning melodies from Midi files using an LSTM neural network.

[DonaldAlan]

See README.md for documentation.

[DonaldAlan]

Listen to http://truthsite.org/music/melodies-like-bach.mp3 for some example melodies generated by the neural network. http://truthsite.org/music/melodies-like-pop.mp3 too

[agibsonccc]

I'm wondering if we can turn this in to a datavec pull request for midi files. https://github.com/deeplearning4j/DataVec

[DonaldAlan]

Adam, if I understand what you're suggesting about DataVec, that would be a much larger task this pull request covers. That is, I presume that the task would be to develop a standard, friendly vectorized format for MIDI files. The MIDI standard encompasses lots of options and is low level. I presume that making the format friendly would be a lot of work. And different users would be interested in different features of MIDI. Is this what you had in mind?

http://musedata.org/ describes some high level symbolic formats for representing music.

[agibsonccc]

Right I'm just wondering how much of this we could turn in to a custom record reader. I'm not sure what you mean by "friendly format". It sounds like you are interpreting I am asking you to rewrite the MIDI spec or something o_0. All datavec does is converts raw data in to ndarrays using an underlying parsing library. I know when I implement any recordreader there is another parsing library under it.

[DonaldAlan]

If the ask is to encode MIDI messages into datavec arrays without much abstraction or feature extraction, it's probably not hard. (Music from Scratch for Midi?) But I suspect that abstracting MIDI to a higher level format might make learning a lot easier. When I say MIDI is low level, here's an example: a MIDI file has a NoteOn event for a particular channel and pitch, followed, at some later time, by the corresponding NoteOff event. The two events are connected to each other only implicitly, via being for the same pitch and channeI. MIDI doesn't directly have the notion of quarter notes or whole notes, Also, interspersed at random times are MIDI events to change instruments, channel pressure, polyphonic key pressure, etc.

[agibsonccc]

Those just sound like configuration values to me. The point of the record reader is to take in a configuration and return what you tell it to. It's not meant to be something where you "create" a format. If we did it for opencv we can do it for audio.

[DonaldAlan]

The google Magenta project (https://magenta.tensorflow.org/2016/06/10/recurrent-neural-network-generation-tutorial/) has code "that converts MIDI files to a format that TensorFlow can understand, making it easy to create training datasets from any collection of MIDI files." Digging into it I see "we are providing code that reads an archive of MIDI files and outputs monophonic melody lines extracted from them in a format TensorFlow can understand. " That sounds a lot like what I did, independently.

If I plan to take this further, we could possibly adopt some of their formats.

I found in https://github.com/tensorflow/magenta/tree/master/magenta/models/melody_rnn that "the first step will be to convert a collection of MIDI files into NoteSequences. NoteSequences are protocol buffers, which is a fast and efficient data format, and easier to work with than MIDI files" By coincidence, I too chose the name NoteSequence.

I looked at their protobuf format for MIDI NoteSequences, and it stores PitchBend, ControlChange, velocity, and other low level MIDI events, unlike my code. But magenta does preprocess/abstract the MIDI events in the way I do and suggested: it represents Notes as having a start time and end time.

[AlexDBlack]

At first glance: this looks quite cool (and thanks for the samples)

I agree with @agibsonccc that having the midi-specific sequence record reader in DataVec would be better (reusable, maintainable, unit tested, etc). For example, if I was doing the character examples again today, I'd use a DataVec sequence record reader (and, will likely convert the character example data pipeline to that, at some point in the future).

For a RNN, what that ultimately looks like in practice: a List<List<Writable>> for each sequence (at least until we move to binary DataVec records in the future). Basically, the inner lists are values within each time step, the outer list is over time steps.
The record reader has to do the mapping from midi to that (with an option to append the feature vector shifted by 1, to give us our labels).

[DonaldAlan]

Perhaps the question is: should we release this simple example as is (and develop a fuller implementation for a later release)? Or wait til it's more complete (datavec, etc)?

It seems to me that most of the other examples in dl4-examples are simple demos, not complete solutions. Already, the fact that I process MIDI files makes it nontrivial.

Obviously, my preference is to release this in stages: first this simple demo (cleaned up, if necessary), later a more complete implementation.

[DonaldAlan]

At some point I'm gonna make an interactive app that lets you "prime" the neural network with a melody, similar to generationInitialization in GravesLSTMCharModellingExample. For that purpose I'll have to figure out how to serialize a model, I guess.

[DonaldAlan]

So, do you guys want to hold off on merging this until it incorporates DataVec? Or is it valuable enough as is?

[DonaldAlan]

I checked in the gradients visualization code that used 3d JavaFX. I improved its implementation so that now it works with any MultilLayerNetwork, sampling from the various dimensions of the gradient INDArray.

It wouldn't be hard to make it work for a ComputationGraph. I can't do everything at once.

[agibsonccc]

Could we get the merge conflicts cleaned up?

[huitseeker]

Would you mind rebasing this ? for you this would amount to :

git checkout master
git remote add github https://github.com/deeplearning4j/dl4j-examples.git
git remote update
git rebase github/master

[agibsonccc]

I'll be ok with this once we run formatting and clean up the git comments.

[agibsonccc]

Could we get rid of the git comments here?

[agibsonccc]

Same thing - git comments.

[DonaldAlan]

I reformatted, removed the git comments, and updated the documentation.

[DonaldAlan]

Now that this is merged, if I want to make updates, how should I do it? I presume I shouldn't use the same branch in my fork (DonaldAlan). Should I use a new branch from github/master? A new fork?

[agibsonccc]

New branch is fine.

[DonaldAlan]

New branch to fix a minor bug and update documentation: #432