New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MELOD4J package for learning melodies from Midi files using an LSTM neural network. #421

Merged
merged 24 commits into from Apr 8, 2017

Conversation

Projects
None yet
4 participants
@DonaldAlan
Contributor

DonaldAlan commented Mar 25, 2017

See README.md for documentation.

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Mar 26, 2017

Contributor

Listen to http://truthsite.org/music/melodies-like-bach.mp3 for some example melodies generated by the neural network. http://truthsite.org/music/melodies-like-pop.mp3 too

Contributor

DonaldAlan commented Mar 26, 2017

Listen to http://truthsite.org/music/melodies-like-bach.mp3 for some example melodies generated by the neural network. http://truthsite.org/music/melodies-like-pop.mp3 too

@agibsonccc

This comment has been minimized.

Show comment
Hide comment
@agibsonccc

agibsonccc Mar 26, 2017

Member

I'm wondering if we can turn this in to a datavec pull request for midi files. https://github.com/deeplearning4j/DataVec

Member

agibsonccc commented Mar 26, 2017

I'm wondering if we can turn this in to a datavec pull request for midi files. https://github.com/deeplearning4j/DataVec

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Mar 26, 2017

Contributor

Adam, if I understand what you're suggesting about DataVec, that would be a much larger task this pull request covers. That is, I presume that the task would be to develop a standard, friendly vectorized format for MIDI files. The MIDI standard encompasses lots of options and is low level. I presume that making the format friendly would be a lot of work. And different users would be interested in different features of MIDI. Is this what you had in mind?

http://musedata.org/ describes some high level symbolic formats for representing music.

Contributor

DonaldAlan commented Mar 26, 2017

Adam, if I understand what you're suggesting about DataVec, that would be a much larger task this pull request covers. That is, I presume that the task would be to develop a standard, friendly vectorized format for MIDI files. The MIDI standard encompasses lots of options and is low level. I presume that making the format friendly would be a lot of work. And different users would be interested in different features of MIDI. Is this what you had in mind?

http://musedata.org/ describes some high level symbolic formats for representing music.

@agibsonccc

This comment has been minimized.

Show comment
Hide comment
@agibsonccc

agibsonccc Mar 26, 2017

Member

Right I'm just wondering how much of this we could turn in to a custom record reader. I'm not sure what you mean by "friendly format". It sounds like you are interpreting I am asking you to rewrite the MIDI spec or something o_0. All datavec does is converts raw data in to ndarrays using an underlying parsing library. I know when I implement any recordreader there is another parsing library under it.

Member

agibsonccc commented Mar 26, 2017

Right I'm just wondering how much of this we could turn in to a custom record reader. I'm not sure what you mean by "friendly format". It sounds like you are interpreting I am asking you to rewrite the MIDI spec or something o_0. All datavec does is converts raw data in to ndarrays using an underlying parsing library. I know when I implement any recordreader there is another parsing library under it.

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Mar 26, 2017

Contributor

If the ask is to encode MIDI messages into datavec arrays without much abstraction or feature extraction, it's probably not hard. (Music from Scratch for Midi?) But I suspect that abstracting MIDI to a higher level format might make learning a lot easier. When I say MIDI is low level, here's an example: a MIDI file has a NoteOn event for a particular channel and pitch, followed, at some later time, by the corresponding NoteOff event. The two events are connected to each other only implicitly, via being for the same pitch and channeI. MIDI doesn't directly have the notion of quarter notes or whole notes, Also, interspersed at random times are MIDI events to change instruments, channel pressure, polyphonic key pressure, etc.

Contributor

DonaldAlan commented Mar 26, 2017

If the ask is to encode MIDI messages into datavec arrays without much abstraction or feature extraction, it's probably not hard. (Music from Scratch for Midi?) But I suspect that abstracting MIDI to a higher level format might make learning a lot easier. When I say MIDI is low level, here's an example: a MIDI file has a NoteOn event for a particular channel and pitch, followed, at some later time, by the corresponding NoteOff event. The two events are connected to each other only implicitly, via being for the same pitch and channeI. MIDI doesn't directly have the notion of quarter notes or whole notes, Also, interspersed at random times are MIDI events to change instruments, channel pressure, polyphonic key pressure, etc.

@agibsonccc

This comment has been minimized.

Show comment
Hide comment
@agibsonccc

agibsonccc Mar 26, 2017

Member

Those just sound like configuration values to me. The point of the record reader is to take in a configuration and return what you tell it to. It's not meant to be something where you "create" a format. If we did it for opencv we can do it for audio.

Member

agibsonccc commented Mar 26, 2017

Those just sound like configuration values to me. The point of the record reader is to take in a configuration and return what you tell it to. It's not meant to be something where you "create" a format. If we did it for opencv we can do it for audio.

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Mar 26, 2017

Contributor

The google Magenta project (https://magenta.tensorflow.org/2016/06/10/recurrent-neural-network-generation-tutorial/) has code "that converts MIDI files to a format that TensorFlow can understand, making it easy to create training datasets from any collection of MIDI files." Digging into it I see "we are providing code that reads an archive of MIDI files and outputs monophonic melody lines extracted from them in a format TensorFlow can understand. " That sounds a lot like what I did, independently.

If I plan to take this further, we could possibly adopt some of their formats.

I found in https://github.com/tensorflow/magenta/tree/master/magenta/models/melody_rnn that "the first step will be to convert a collection of MIDI files into NoteSequences. NoteSequences are protocol buffers, which is a fast and efficient data format, and easier to work with than MIDI files" By coincidence, I too chose the name NoteSequence.

I looked at their protobuf format for MIDI NoteSequences, and it stores PitchBend, ControlChange, velocity, and other low level MIDI events, unlike my code. But magenta does preprocess/abstract the MIDI events in the way I do and suggested: it represents Notes as having a start time and end time.

Contributor

DonaldAlan commented Mar 26, 2017

The google Magenta project (https://magenta.tensorflow.org/2016/06/10/recurrent-neural-network-generation-tutorial/) has code "that converts MIDI files to a format that TensorFlow can understand, making it easy to create training datasets from any collection of MIDI files." Digging into it I see "we are providing code that reads an archive of MIDI files and outputs monophonic melody lines extracted from them in a format TensorFlow can understand. " That sounds a lot like what I did, independently.

If I plan to take this further, we could possibly adopt some of their formats.

I found in https://github.com/tensorflow/magenta/tree/master/magenta/models/melody_rnn that "the first step will be to convert a collection of MIDI files into NoteSequences. NoteSequences are protocol buffers, which is a fast and efficient data format, and easier to work with than MIDI files" By coincidence, I too chose the name NoteSequence.

I looked at their protobuf format for MIDI NoteSequences, and it stores PitchBend, ControlChange, velocity, and other low level MIDI events, unlike my code. But magenta does preprocess/abstract the MIDI events in the way I do and suggested: it represents Notes as having a start time and end time.

@AlexDBlack

This comment has been minimized.

Show comment
Hide comment
@AlexDBlack

AlexDBlack Mar 26, 2017

Member

At first glance: this looks quite cool (and thanks for the samples)

I agree with @agibsonccc that having the midi-specific sequence record reader in DataVec would be better (reusable, maintainable, unit tested, etc). For example, if I was doing the character examples again today, I'd use a DataVec sequence record reader (and, will likely convert the character example data pipeline to that, at some point in the future).

For a RNN, what that ultimately looks like in practice: a List<List<Writable>> for each sequence (at least until we move to binary DataVec records in the future). Basically, the inner lists are values within each time step, the outer list is over time steps.
The record reader has to do the mapping from midi to that (with an option to append the feature vector shifted by 1, to give us our labels).

Member

AlexDBlack commented Mar 26, 2017

At first glance: this looks quite cool (and thanks for the samples)

I agree with @agibsonccc that having the midi-specific sequence record reader in DataVec would be better (reusable, maintainable, unit tested, etc). For example, if I was doing the character examples again today, I'd use a DataVec sequence record reader (and, will likely convert the character example data pipeline to that, at some point in the future).

For a RNN, what that ultimately looks like in practice: a List<List<Writable>> for each sequence (at least until we move to binary DataVec records in the future). Basically, the inner lists are values within each time step, the outer list is over time steps.
The record reader has to do the mapping from midi to that (with an option to append the feature vector shifted by 1, to give us our labels).

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Mar 26, 2017

Contributor

Perhaps the question is: should we release this simple example as is (and develop a fuller implementation for a later release)? Or wait til it's more complete (datavec, etc)?

It seems to me that most of the other examples in dl4-examples are simple demos, not complete solutions. Already, the fact that I process MIDI files makes it nontrivial.

Obviously, my preference is to release this in stages: first this simple demo (cleaned up, if necessary), later a more complete implementation.

Contributor

DonaldAlan commented Mar 26, 2017

Perhaps the question is: should we release this simple example as is (and develop a fuller implementation for a later release)? Or wait til it's more complete (datavec, etc)?

It seems to me that most of the other examples in dl4-examples are simple demos, not complete solutions. Already, the fact that I process MIDI files makes it nontrivial.

Obviously, my preference is to release this in stages: first this simple demo (cleaned up, if necessary), later a more complete implementation.

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Mar 27, 2017

Contributor

At some point I'm gonna make an interactive app that lets you "prime" the neural network with a melody, similar to generationInitialization in GravesLSTMCharModellingExample. For that purpose I'll have to figure out how to serialize a model, I guess.

Contributor

DonaldAlan commented Mar 27, 2017

At some point I'm gonna make an interactive app that lets you "prime" the neural network with a melody, similar to generationInitialization in GravesLSTMCharModellingExample. For that purpose I'll have to figure out how to serialize a model, I guess.

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Mar 28, 2017

Contributor

So, do you guys want to hold off on merging this until it incorporates DataVec? Or is it valuable enough as is?

Contributor

DonaldAlan commented Mar 28, 2017

So, do you guys want to hold off on merging this until it incorporates DataVec? Or is it valuable enough as is?

@DonaldAlan DonaldAlan changed the title from Initial checkin of melodl4j package for learning melodies from Midi files using an LSTM neural network. to MELOD4J package for learning melodies from Midi files using an LSTM neural network. Mar 30, 2017

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Mar 31, 2017

Contributor

I checked in the gradients visualization code that used 3d JavaFX. I improved its implementation so that now it works with any MultilLayerNetwork, sampling from the various dimensions of the gradient INDArray.

It wouldn't be hard to make it work for a ComputationGraph. I can't do everything at once.

Contributor

DonaldAlan commented Mar 31, 2017

I checked in the gradients visualization code that used 3d JavaFX. I improved its implementation so that now it works with any MultilLayerNetwork, sampling from the various dimensions of the gradient INDArray.

It wouldn't be hard to make it work for a ComputationGraph. I can't do everything at once.

@agibsonccc

This comment has been minimized.

Show comment
Hide comment
@agibsonccc

agibsonccc Apr 8, 2017

Member

Could we get the merge conflicts cleaned up?

Member

agibsonccc commented Apr 8, 2017

Could we get the merge conflicts cleaned up?

@huitseeker

This comment has been minimized.

Show comment
Hide comment
@huitseeker

huitseeker Apr 8, 2017

Contributor

Would you mind rebasing this ? for you this would amount to :

git checkout master
git remote add github https://github.com/deeplearning4j/dl4j-examples.git
git remote update
git rebase github/master
Contributor

huitseeker commented Apr 8, 2017

Would you mind rebasing this ? for you this would amount to :

git checkout master
git remote add github https://github.com/deeplearning4j/dl4j-examples.git
git remote update
git rebase github/master
@agibsonccc

I'll be ok with this once we run formatting and clean up the git comments.

Show outdated Hide outdated README.md Outdated
Show outdated Hide outdated README.md Outdated
@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Apr 8, 2017

Contributor

I reformatted, removed the git comments, and updated the documentation.

Contributor

DonaldAlan commented Apr 8, 2017

I reformatted, removed the git comments, and updated the documentation.

@agibsonccc agibsonccc merged commit 19e9464 into deeplearning4j:master Apr 8, 2017

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Apr 8, 2017

Contributor

Now that this is merged, if I want to make updates, how should I do it? I presume I shouldn't use the same branch in my fork (DonaldAlan). Should I use a new branch from github/master? A new fork?

Contributor

DonaldAlan commented Apr 8, 2017

Now that this is merged, if I want to make updates, how should I do it? I presume I shouldn't use the same branch in my fork (DonaldAlan). Should I use a new branch from github/master? A new fork?

@agibsonccc

This comment has been minimized.

Show comment
Hide comment
@agibsonccc

agibsonccc Apr 8, 2017

Member

New branch is fine.

Member

agibsonccc commented Apr 8, 2017

New branch is fine.

@DonaldAlan

This comment has been minimized.

Show comment
Hide comment
@DonaldAlan

DonaldAlan Apr 8, 2017

Contributor

New branch to fix a minor bug and update documentation: #432

Contributor

DonaldAlan commented Apr 8, 2017

New branch to fix a minor bug and update documentation: #432

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment