Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moodbar: Find better music representation #99

Closed
sjohannes opened this issue May 29, 2015 · 11 comments
Closed

Moodbar: Find better music representation #99

sjohannes opened this issue May 29, 2015 · 11 comments
Assignees
Milestone

Comments

@sjohannes
Copy link
Member

@sjohannes sjohannes commented May 29, 2015

Currently Moodbar is based on the audio spectrum; it maps low frequencies to red, medium frequencies to green, and high frequencies to blue.

I feel that using a different audio feature can give us a better representation of musical segments. I've been experimenting with MFCC, and I think it produces a more useful output. Unfortunately the library I'm using, Essentia, is not packaged in Debian/Ubuntu, so I'm thinking to switch to Aubio.

Furthermore, using RGB for this just doesn't seem right. I'd like to explore using a different colourspace, e.g. Y'UV, mapping loudness to the luma and some other audio feature to the chroma.

There are probably research out there dealing with this very issue, so it may be better to do a bit of literature review first.

@sjohannes sjohannes self-assigned this May 29, 2015
@sjohannes sjohannes added this to the Someday milestone May 29, 2015
@bitterjug

This comment has been minimized.

Copy link
Contributor

@bitterjug bitterjug commented May 29, 2015

I'm thinking of reworkinm moodbar as a gtk widget. I currently use the
waveform view option of moodbar just giving a sense of amplitude . it'd be
nice to make the visualization configurable.
On 29 May 2015 12:32, "Johannes Sasongko" notifications@github.com wrote:

Currently Moodbar is based on the audio spectrum; it maps low frequencies
to red, medium frequencies to green, and high frequencies to blue.

I feel that using a different audio feature can give us a better
representation of musical segments. I've been experimenting with MFCC, and
I think it produces a more useful output. Unfortunately the library I'm
using, Essentia, is not packaged in Debian/Ubuntu, so I'm thinking to
switch to Aubio.

Furthermore, using RGB for this just doesn't seem right. I'd like to
explore using a different colourspace, e.g. Y'UV, mapping loudness to the
luma and some other audio feature to the chroma.

There are probably research out there dealing with this very issue, so it
may be better to do a bit of literature review first.


Reply to this email directly or view it on GitHub
#99.

@sjohannes

This comment has been minimized.

Copy link
Member Author

@sjohannes sjohannes commented May 30, 2015

Note that I'm rewriting the rendering side of the Moodbar plugin for GTK+ 3 in the new-moodbar branch. At the same time I'm trying to make it easier to use the moodbar as its own widget.

@genodeftest

This comment has been minimized.

Copy link
Contributor

@genodeftest genodeftest commented Nov 27, 2015

How is your progress on the new moodbar implementation? Shall I report bugs if I find some?

@sjohannes

This comment has been minimized.

Copy link
Member Author

@sjohannes sjohannes commented Nov 28, 2015

The rewrite is done, but some features are not implemented (yet), compared to the previous version.

@genodeftest

This comment has been minimized.

Copy link
Contributor

@genodeftest genodeftest commented Nov 28, 2015

Ok, because

  1. I don't see any content at all, just a grey, empty frame.
  2. I sometimes get tracebacks
@sjohannes

This comment has been minimized.

Copy link
Member Author

@sjohannes sjohannes commented Nov 28, 2015

Please file a report for it; the basic moodbar functionality should work as long as you have moodbar installed.

@slmeyer

This comment has been minimized.

Copy link

@slmeyer slmeyer commented Apr 21, 2016

I have come to rely on the moodbar waveform amplitude representation, really shows where in the music you are.

@Hackswell

This comment has been minimized.

Copy link

@Hackswell Hackswell commented Apr 26, 2016

@genodeftest : I had the same problem with Kubuntu vivid. As sjohannes said, install the "moodbar" package for your distro. Under Kubuntu 16.04, moodbar is installed by default, and it's back!

@genodeftest

This comment has been minimized.

Copy link
Contributor

@genodeftest genodeftest commented May 16, 2017

Music representation:

How about using the HSV or HSL color spaces? they seem to fit better here. For HSV, "white noise" will become grey, for HSL it will become white. Narrow-band music will become a very saturated color.

  1. split up sound file into small blocks, so you get a block(time) mapping
  2. fourier-transform each block (maybe using MFCC), so you get a intensity(frequency) mapping. Limit the number of frequencies you get, e.g. by merging into 0.1 kHz wide blocks
  3. Map the audible frequency to a wavelength ("hue") of the visible color spectrum. Could use the HSL (lightness: 0.5) or HSV (value: 1) model, with saturation of 1. Now we have a intensity(hue) mapping.
  4. Calculate the weighted average on each block. The intensity is used for weighting. In HSL/HSV model, the color would be gray in case of white noise and a saturated color in case of narrow-band music. Now we have a single color on each block, i.e. a color(time) mapping.
  5. transform to RGB color space.
  6. calculate a volume(time) mapping, i.e. the volume of each block.
  7. draw on a widget. x=time, y=volume, color=color
@Sur3

This comment has been minimized.

Copy link

@Sur3 Sur3 commented May 2, 2018

How about a full spectrum instead of the compression to a single color in every timestep, meaning to also separate the vertical bars in different frequencies to get a real spectrum..?!?

@sjohannes

This comment has been minimized.

Copy link
Member Author

@sjohannes sjohannes commented May 2, 2018

That's an interesting idea, essentially using a spectogram as the seekbar. If anyone is interested in implementing this, it wouldn't be too difficult to integrate it with the Moodbar plugin, and I'd be happy to help answer questions and review changes.

However, I'm going to close this ticket because I never quite managed to do the things I set out to do in the initial comment. The MFCC experiment I did was promising but in the end I wasn't able to find a suitable library (widely packaged, fast, supports all formats that Exaile supports) to implement it in. I've played around with the other ideas as well but nothing really interesting came out of it.

I'm willing to reopen this if there's something concrete happening, but at the moment this issue is not something I'm planning to work on.

To anyone following this bug for the waveform mode feature, don't worry! It's still in my todo list and is now tracked in issue #499.

@sjohannes sjohannes closed this May 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.