The project aims studying the audio signal in terms of its perceptual characteristics, resulting in an algorithm that will be able to detect (map) unknown audio snippets from a large database of known songs.
C# Other
Latest commit 8180c3d Dec 6, 2016 @AddictedCS committed on GitHub Typo fix in readme

README.md

Audio fingerprinting and recognition in .NET

Join the chat at https://gitter.im/soundfingerprinting/Lobby

soundfingerprinting is a C# framework designed for developers, enthusiasts, researchers in the fields of audio and digital signal processing, data mining and audio recognition. It implements an efficient algorithm which provides fast insert and retrieval of acoustic fingerprints with high precision and recall rate.

Build Status

Documentation

Below code snippet shows how to extract acoustic fingerprints from an audio file and later use them as identifiers to recognize unknown audio query. These sub-fingerprints (or fingerprints, 2 terms are used interchangeably) will be stored in a configurable backend. The interfaces for fingerprinting and querying audio files are implemented as Fluent Interfaces using Builder and Command patterns.

private readonly IModelService modelService = new InMemoryModelService(); // store fingerprints in RAM
private readonly IAudioService audioService = new NAudioService(); // use NAudio audio processing library
private readonly IFingerprintCommandBuilder fingerprintCommandBuilder = new FingerprintCommandBuilder();

public void StoreAudioFileFingerprintsInStorageForLaterRetrieval(string pathToAudioFile)
{
    var track = new TrackData("GBBKS1200164", "Adele", "Skyfall", "Skyfall", 2012, 290);

    // store track metadata in the datasource
    var trackReference = modelService.InsertTrack(track);

    // create hashed fingerprints
    var hashedFingerprints = fingerprintCommandBuilder
                                .BuildFingerprintCommand()
                                .From(pathToAudioFile)
                                .UsingServices(audioService)
                                .Hash()
                                .Result;

    // store hashes in the database for later retrieval
    modelService.InsertHashDataForTrack(hashedFingerprints, trackReference);
}

The default storage, which comes bundled with soundfingerprinting package, is a plain RAM storage, managed by InMemoryModelService. The following list of persistent storages is available for general use:

Once you've inserted the fingerprints into the datastore, later you might want to query the storage in order to recognize the song those samples you have. The origin of query samples may vary: file, URL, microphone, radio tuner, etc. It's up to your application, where you get the samples from.

private readonly IQueryCommandBuilder queryCommandBuilder = new QueryCommandBuilder();

public TrackData GetBestMatchForSong(string queryAudioFile)
{
    int secondsToAnalyze = 10; // number of seconds to analyze from query file
    int startAtSecond = 0; // start at the begining

    // query the underlying database for similar audio sub-fingerprints
    var queryResult = queryCommandBuilder.BuildQueryCommand()
                                         .From(queryAudioFile, secondsToAnalyze, startAtSecond)
                                         .UsingServices(modelService, audioService)
                                         .Query()
                                         .Result;
    if(queryResult.ContainsMatches)
    {
        return queryResult.BestMatch.Track; // successful match has been found
    }

    return null; // no match has been found
}

Every ResultEntry object will contain the following information:

  • Track - matched track from the datastore
  • QueryMatchLength - returns how many query seconds matched the resulting track
  • TrackStartsAt - returns where does the matched track starts, always relative to the query
  • Coverage - returns a value between [0, 1], informing how much the query covered the resulting track (i.e. a 2 minutes query found a 30 seconds track within it, starting at 100th second, coverage will be equal to (120 - 100)/30 ~= 0.66)
  • Confidence - returns a value between [0, 1]. A value below 0.15 is most probably a false positive. A value bigger than 0.15 is very likely to be an exact match. For good audio quality queries you can expect getting a confidence > 0.5.

Upgrade from 2.x to 3.x

All users of soundfingerprinting are encouraged to migrate to v3.x due to all sorts of important bug-fixes and improvements. Version 3.0.0 is faster, more accurate, and provides an intuitive response interface with additional information about the query and the match. When migrating make sure to re-insert the fingerprints into the datasource, since their internal signature changed slightly.

List of additional soundfingerprinting integrations

  • SoundFingerprinting.Audio.Bass - Bass.Net audio library integration, comes as a replacement for NAudio default service. Works faster, more accurate resampling, supports multiple audio formats, independent upon target OS. Bass is free for non-comercial use.
  • All demo apps are now located in separate git repositories, duplicates detector, sound tools.

Algorithm configuration

Fingerprinting and Querying algorithms can be easily parametrized with corresponding configuration objects passed as parameters on command creation.

 var hashDatas = fingerprintCommandBuilder
                           .BuildFingerprintCommand()
                           .From(samples)
                           .WithFingerprintConfig(
                                config =>
                                {
                                    config.Stride = new IncrementalRandomStride(256, 512); // more agressive stride for noisy environments
                                })
                           .UsingServices(audioService)
                           .Hash()
                           .Result;

Each and every configuration parameter can influence the recognition rate, required storage, computational cost, etc. Stick with the defaults, unless you would like to experiment.

The most sensitive parameter (which directly affects precision/recall rate) is Stride parameter. Empirically it was determined that using a smaller stride during querying gives a better recall rate, at the expense of execution time.

In case you need directions for fine-tunning the algorithm for your particular use case do not hesitate to contact me.

Third party dependencies

Links to the third party libraries used by soundfingerprinting project.

FAQ

  • Can I apply this algorithm for speech recognition purposes? > No. The granularity of one fingerprint is roughly ~1.46 seconds.
  • Can the algorithm detect exact query position in resulted track? > Yes.
  • Can I use SoundFingerprinting to detect ads in radio streams? > Yes.
  • Will SoundFingerprinting match tracks with samples captured in noisy environment? Yes, but you will have to play around with Stride (decreasing it on both insertion and query) and ThresholdVotes query parameter (decreasing it as well).
  • Can I use SoundFingerprinting framework on Mono? Yes. SoundFingerprinting can be used in cross-platform applications. Just keep in mind that the default audio service NAudio, requires Windows native DLLs. Since these are not available in Unix, you can use the override method which asks for AudioSamples as the source for fingerprinting and querying. It's the responsability of the caller to provide mono audio samples at 5512 frequency rate. If this condition is met, the algorithm will not invoke any methods from NAudio.

Binaries

git clone git@github.com:AddictedCS/soundfingerprinting.git

In order to build latest version of the SoundFingerprinting assembly run the following command from repository root.

.\build.cmd

Get it on NuGet

Install-Package SoundFingerprinting

Demo

My description of the algorithm alogside with the demo project can be found on CodeProject The demo project is a Audio File Duplicates Detector. Its latest source code can be found here. Its a WPF MVVM project that uses the algorithm to detect what files are perceptually very similar.

Contribute

If you want to contribute you are welcome to open issues or discuss on issues page. Feel free to contact me for any remarks, ideas, bug reports etc.

License

The framework is provided under MIT license agreement. The theoretical description of the algorithm can be read in Content Fingerprinting using Wavelets paper.

Special thanks to JetBrains for providing this project with a license for ReSharper!

JetBrains

© Soundfingerprinting, 2010-2016, ciumac.sergiu@gmail.com