Skip to content
Java library for fetching and parsing rekordbox exports and track analysis files.
Java RPC
Branch: master
Clone or download
Latest commit d6d3674 Aug 25, 2019

README.adoc

Crate Digger

A Java library for fetching and parsing rekordbox media exports and track analysis files.

License Eclipse%20Public%20License%201.0 blue

This project uses the Kaitai Struct compiler with the help of a Maven plugin to create classes that can parse and output binary data structures in a convenient and efficient way. It uses them to create Java classes, because it was created to support Beat Link, but other projects can use them to output code for other languages.

It also uses the jrpcgen tool (which is part of the Remote Tea project), via another plugin, to generate classes that know how to talk to the non-standard NFSv2 file servers running in Link-capable players, so the rekordbox data can be reliably obtained even during big shows where all four players are in use.

PDB Database

The file rekordbox_pdb.ksy contains the structure definitions needed to parse exported rekordbox databases (export.pdb files).

ℹ️
Huge thanks to Fabian Lesniak for figuring out the details of how to interpret these files in his python-prodj-link project and Mikhail Yakshin for helping me quickly learn the more subtle aspects of Kaitai Struct. And this was all started by a question Evan Purkhiser posted on Stack Exchange.

There is a Structure Analysis article describing the details of what we have learned about these file formats. Reading that will help make sense of the exploration tools and the objects returned by this library.

Exploring the Analysis

One of the amazingly cool things about Kaitai Struct is that you can use its Web IDE to see how the structure definitions work and visually explore the contents of files you are analyzing. This also means you can look inside your own .pdb files and check my work, or get a better understanding of how to use the generated parsers. To do that, simply upload the .pdb file you want to examine to the Web IDE (it doesn’t actually go to the web, it just gets put in your local browser storage), then also upload my rekordbox_pdb.ksy file, and the Web IDE will parse the exported database, letting you explore the structures in the tree view, and see the corresponding raw bytes in the hex viewer.

💡
You can find the export.pdb file on a media stick prepared by rekordbox inside the PIONEER folder, which may be invisible in the Finder, but you can open it using Terminal commands if you have to. To download my rekordbox_pdb.ksy, click on its link, then click on the Raw button in the header above the first line of the listing, then tell your browser to save it to disk. Be sure to keep the .ksy extension. Then you can upload it to the Kaitai Struct Web IDE.

ANLZ Data

Each track in a rekordbox database also has ANLZnnnn.DAT and ANLZnnnn.EXT files associated with it, containing the beat grid, an index allowing rapid seeking to any time in variable-bit rate audio files, the waveforms, memory cues and loop points. The paths to these files are found inside the corresponding track record.

The structure definitions for these files are in rekordbox_anlz.ksy. You can use it with the Kaitai Struct Web IDE as described above to explore analysis files found in your own exported media.

Using the Library

Crate Digger is available through Maven Central, so to use it in your Maven project, all you need is to include the appropriate dependency.

Maven Central

Click the maven central badge above to view the repository entry for crate-digger. The proper format for including the latest release as a dependency in a variety of tools, including Leiningen if you are using beat-link from Clojure, can be found in the Dependency Information section.

There are two halves to what Crate Digger offers. The first is an ability to talk to the nonstandard Network File System servers that are running in Pioneer players, and ask them to deliver the rekordbox data export and track analysis files (programs like Beat Link need these files to provide smooth integrations with the music being performed). The second half is the ability to parse the contents of those files, as described above.

Retreiving Files

The class org.deepsymmetry.cratedigger.FileFetcher is a singleton, so to work with it you will start by calling getInstance(), as is customary. Retrieving a file is then as simple as this:

FileFetcher fetcher = FileFetcher.getInstance();
fetcher.fetch(playerAddress, mountPath, sourcePath, destination);

playerAddress is an InetAddress object holding the address of the player from which you want to download a file. mountPath identifies the media slot you want to get information from, as shown in the table below. sourcePath is the path to the specific file you want within the mounted media, and destination is a File object identifying where you want the downloaded data to be stored.

Table 1. Media Slot Mount Paths

Media Slot

Mount Path

SD

/B/

USB

/C/

❗️

The FileFetcher caches information about players to make requests more efficient, so it is important for you to tell it when a player goes away, or unmounts one of its media slots, by calling:

fetcher.removePlayer(playerAddress);

Parsing Structures

The class org.deepsymmetry.cratedigger.Database provides support for accessing the contents of rekordbox database export files. You can create an instance to wrap a File instance that contains such an export (for example one that you downloaded using the fetch method above). Then you can query it for track and other information:

Database database = new Database(downloadedFile);
RekordboxPdb.TrackRow track = database.findTrack(1);
System.out.println(database.getText(track.title()));

Strings (like titles, artist names, etc.) are represented by a variety of structures with different encodings, so a getText() method is provided to convert them into ordinary Java strings.

See the API documentation for more details about these classes, and the Structure Analysis for more details about the file formats.

Logging

Crate Digger uses slf4j to allow you to integrate it with whatever Java logging framework your project is using, so you will need to include the appropriate slf4j binding on your class path.

Unfinished Tasks

  • There are still more tables to be figured out. Columns looks like the list of things that can be searched by, so perhaps it will hold some clues for how to find and use the index tables, which must exist because it would be horribly slow for the players to do a linear scan through the main sparse tables whenever they wanted a record.

  • If we could figure out how to use the indices ourselves, we could avoid having to load the whole file and index it ourselves.

Building the source

As noted above, he Maven project uses a plugin to run the the jrpcgen tool (which is part of the Remote Tea project) to generate Java classes to implement the ONC RPC specifications found in src/main/rpc. (These are used for communicating with the NFS servers in CDJs.) It also uses the Kaitai Struct Compiler through another plugin to generate Java classes that can parse the rekordbox databases it downloads from the players, based on the specifications found in src/main/kaitai.

These things happen for you automatically during the code generation phase of the Maven build. If you want to use something other than Maven, you will need to figure out how to configure and run the tools yourself.

Building the Structure Analysis

The source is found in doc/Analysis.tex. I started out using pdfLaTeX to write and format the document, but then, at the recommendation of one of the Kaitai Struct developers, switched to XeLaTeX in order to take advantage of newer fonts, Unicode, and ligatures. I tried setting up the document preamble so that both processors will still work, although the line breaks have now been optimized for XeLaTex, and if you want to use that you will either need to have the Garamond Premier Pro and Andale Mono font packages available on your system, or choose different font families that you actually do own in the \ifxetex section at the top.

The easiest way to get all the TeX and LaTeX pieces you need is probably via TeX Live, which is how I obtained it when I started working on the protocol analysis document for dysentery.

Contributing

If you have ideas, discoveries, or even code you’d like to share, that’s fantastic! Please take a look at the guidelines and get in touch!

Licenses

Deep Symmetry logo Copyright © 2018–2019 Deep Symmetry, LLC

Distributed under the Eclipse Public License 1.0. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software. A copy of the license can be found in LICENSE within this project.

Library and Build Tool Licenses

The Kaitai Struct Compiler is licensed under the GNU General Public License, version 3 and the Kaitai Java runtime embedded in crate-digger is licensed under the MIT License.

You can’t perform that action at this time.