New fields in Serato DJ 1.6.x

[VxJasonxV]

It looks like Serato updated their session database format at some point. I was poking around with my files and just noticed this huge spew of messages at the top after -d'ing a session file:

$ php ~/Development/sslscrobbler/historyreader.php -d ~/Music/_Serato_/History/Sessions/2580.session
2014-06-27 08:15:17 INFO: SSLFileReader - reading structure of /Users/jason/Music/_Serato_/History/Sessions/2580.session...
2014-06-27 08:15:17 INFO: SSLFileReader - asking structure DOM to parse /Users/jason/Music/_Serato_/History/Sessions/2580.session...
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1Unknown field 71: 4073 cdd8 4b3b 8f27                                                             @s!!K;!'
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1Unknown field 71: 4072 2bd7 0a3d 70a4                                                             @r+!A=p!
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 00                                                                              .
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1

Warning: vsprintf(): Too few arguments in /Users/jason/Development/sslscrobbler/SSL/Logger/L.php on line 93
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker -
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1
2014-06-27 08:15:17 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1Unknown field 71: 4079 d999 9999 999a                                                             @y!!!!!!
2014-06-27 08:15:17 INFO: SSLFileReader - printing structure of /Users/jason/Music/_Serato_/History/Sessions/2580.session...
CHUNK<vrsn>:
    version => 1.0/Serato Scratch LIVE Review

[ben-xo]

I'm curious what caused that vsprintf() warning.

Yeah they seem to add new fields from time to time, and they mostly seem to be fields from tracks (e.g. things like key, remixer, stuff to do with video, maybe stuff that is only relevant to non-mp3s etc etc.) I haven't personally seen anything with fields 70 or 71 though. If you can find out what track or tracks they correspond to, there might be some clues in the ID3 (or other metadata) tags as to what those fields are.

'00' or '01' is obviously some sort of simple "on" / "off" for a track.

The other one - 4073 cdd8 4b3b 8f27, etc - no idea. Could be anything, it's a 64-bit number so it could be some sort of timestamp, but it's clearly not critical because only a few tracks have it.

If you fancy it, open up SSLTrackAdat.xoup , add a line like

field71: timestamp. c>unknown71

and see if a field "unknown71" appears and makes sense as a date somewhere in the dump output.

(I know XOUP is a totally ridiculous made up language but what are you going to do.)

[VxJasonxV]

It doesn't seem happy with that. I made that change right above literal data, but now it's outright erroring:

2014-06-27 10:36:19 INFO: SSLFileReader - reading structure of /Users/jason/Music/_Serato_/History/Sessions/2580.session...
2014-06-27 10:36:19 INFO: SSLFileReader - asking structure DOM to parse /Users/jason/Music/_Serato_/History/Sessions/2580.session...
2014-06-27 10:36:19 INFO: XoupCompiler - compiling /Users/jason/Development/sslscrobbler/SSL/Structs/SSLTrackAdat.xoup
2014-06-27 10:36:19 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1
Cannot unpack an odd-sized int of width 19
Try /Users/jason/Development/sslscrobbler/historyreader.php --help

Not all field71's are a 64-bit number?

[ben-xo]

Ah, it's probably a string then. Integers can only have a width (in bytes) of 1, 2, 4 or 8.

try

field71: hex. c>unknown71

and you should get a pretty hexdump, which will at least make it more obvious if it's a string. If they're not constant length, they're usually strings...

[VxJasonxV]

Still no such luck:

2014-06-29 21:15:49 INFO: SSLFileReader - reading structure of /Users/jason/Music/_Serato_/History/Sessions/2580.session...
2014-06-29 21:15:49 INFO: SSLFileReader - asking structure DOM to parse /Users/jason/Music/_Serato_/History/Sessions/2580.session...
2014-06-29 21:15:49 INFO: XOUPSSLTrackAdatUnpacker - Unknown field 70: 01                                                                              1
Cannot unpack an odd-sized int of width 115

Is there any type that will just spit this out raw? Relatedly, is there a way to get a dump of the entire structure with a basic key (number)/value representation, and not just mapped ones?

[ben-xo]

Send me over the session file and I'll have a look.

Ben XO / Last.fm / Bassdrive
http://mixcloud.com/benxo
http://twitter.com/benxo
http://facebook.com/benxomusic
http://www.last.fm/user/ben-xo

[VxJasonxV]

That would be a good idea :). https://gist.github.com/VxJasonxV/2f7139e0c0cb48f51887

[ben-xo]

Welp - when I do "php historyreader.php -d ~/Downloads/2580.session" i get a full structured dump. I was wrong about the UNKNOWN fields appearing though, oops.

If you go to the bottom of SSLTrackAdat.xoup you'll see "known unknowns". Change that block to something like

known unknowns

field16: ascii. c>rUNKNOWN16
field33: ascii. c>rUNKNOWN33
field39: int. c>rUNKNOWN39
field70: int. c>rUNKNOWN70
field71: hex. c>rUNKNOWN71

and they'll all appear in the structured dump.

Unfortunately it's not obvious what any of those unknowns are. (Lots of your fields have UNKNOWN16 and UNKNOWN33 as well).

If you just want to get rid of the warnings, we can just add them to the known unknowns list so they don't elicit a warning.

[rabyunghwa]

Hey dude it seems like there are new fields again since I upgraded to SeratoDJ 1.8.0. Hope this is fixed asap.

[ben-xo]

Cool! - it's not a bug, it's just information that I might find useful at
some point. It won't affect the operation of the program.

You should really look at getting a 64-but PHP though.

On Fri, 13 Nov 2015 at 16:47, ByungHwa Ra notifications@github.com wrote:

Hey dude it seems like there are new fields again since I upgraded to
SeratoDJ 1.8.0. Hope this is fixed asap.
[image: capture]
https://cloud.githubusercontent.com/assets/7411385/11151641/3339d1be-8a69-11e5-8da6-906ce2081f12.PNG

—
Reply to this email directly or view it on GitHub
#17 (comment).

[rabyunghwa]

Okay dude I'll try switching to a 64-bit version! Thank you!

[rabyunghwa]

Just one more thing dude. What does it mean "... passed scrobble point"? Has the song been successfully scrobbled? How do I know if it has been successful?

[rabyunghwa]

Cuz I noticed that the song had been scrobbled to last.fm even tho it said the song passed the scrobble point. So what does it mean exactly?

[ben-xo]

The scrobble point is the minimum length of time you listened to the song
before it's allowed to be scrobbled - more than half way, but no less
than 30 seconds and no more than 5 minutes.

But, the scrobble is not actually submitted until the song has finished
being played, usually after you change tracks (after it goes green in the
history).

On Fri, 13 Nov 2015 at 17:51 ByungHwa Ra notifications@github.com wrote:

Cuz I noticed that the song had been scrobbled to last.fm even tho it
said the song passed the scrobble point. So what does it mean exactly?

—
Reply to this email directly or view it on GitHub
#17 (comment).

[rabyunghwa]

Ohh OK dude thank you so much for such detailed explanation!
On Nov 14, 2015 1:54 AM, "Ben XO" notifications@github.com wrote:

The scrobble point is the minimum length of time you listened to the song
before it's allowed to be scrobbled - more than half way, but no less
than 30 seconds and no more than 5 minutes.

But, the scrobble is not actually submitted until the song has finished
being played, usually after you change tracks (after it goes green in the
history).

On Fri, 13 Nov 2015 at 17:51 ByungHwa Ra notifications@github.com wrote:

Cuz I noticed that the song had been scrobbled to last.fm even tho it
said the song passed the scrobble point. So what does it mean exactly?

—
Reply to this email directly or view it on GitHub
<#17 (comment)
.

—
Reply to this email directly or view it on GitHub
#17 (comment).

[rabyunghwa]

Dude I tried to insert the song into mysql database and it showed one unknown field and I found nothing has been inserted. What's the cause?

$dbConfig = array(
'dsn' => 'mysql:host=localhost;dbname=barsphere',
'user' => 'rabyunghwa',
'pass' => '12345678',
'options' => array(),
'sql' => "UPDATE track SET field=:track, artist=:artist, title=:title, album=:album WHERE         user=:key",
'empty_string' => '<no track is currently playing>',
);

my database name is "barsphere" and the name of the table which i want to insert the song data into is called "track". And the columns in the table are field, artist, title, album and user.

[rabyunghwa]

Here is the mysql cmd prompt: I don't think i'm missing anything here. I inserted that row of data beforehand so that later it can be updated when a new track is being played.

[rabyunghwa]

Okay I got it working now! The reason why it has taken me so long to know how to use this is that I thought that that update statement was the only one to use here. I changed it to an insert statement. And it's working now!

[ben-xo]

Very glad you got it working! :)

I've updated the very latest master to ignore the new fields.

[rabyunghwa]

Okay dude thank you so much! I'm so glad that all the efforts have finally paid off! :)

[rabyunghwa]

Hey dude! Just one more thing! Will this only work with MySQL database? Will it work with Oracle Database as well?

[ben-xo]

I've never tested it with any other database; but because it uses PHP's
"PDO" internally, it should work with any database that you can write the
correct DSN for. PHP does have an Oracle driver, but you'd have to enable
it in the .ini like you did for MySQL.

http://php.net/manual/en/ref.pdo-oci.connection.php

On Sun, 15 Nov 2015 at 13:17, ByungHwa Ra notifications@github.com wrote:

Hey dude! Just one more thing! Will this only work with MySQL database?
Will it work with Oracle Database as well?

—
Reply to this email directly or view it on GitHub
#17 (comment).

[rabyunghwa]

Okay dude thank you very much!

[rabyunghwa]

Hey dude! I've been dealing with Oracle Database and it seems like I'm going the right direction but there is this error

. It says something like "could not resolve connection identifier". I've been googling and stackoverflowing for a while but haven't found anything helpful. Thanks in advance.

[ben-xo]

I'm sorry I really don't know - it could be any number of things. I've
never used this with oracle before. It does sort of sound like either the
DSN is wrong e.g has a typo, or perhaps it's just the wrong choice.

As you are on Windows, you could also try using the ODBC connector. You'll
have to look into setting up a database connection in ODBC system setting
first and giving it a name.

If you look in the config.php file, you'll find you can increase the log
level from INFO to DEBUG and it may give you better clues.
On Tue, 17 Nov 2015 at 08:52, ByungHwa Ra notifications@github.com wrote:

Hey dude! I've been dealing with Oracle Database and it seems like I'm
going the right direction but there is this error
[image: capture]
https://cloud.githubusercontent.com/assets/7411385/11206655/72c45b94-8d4b-11e5-818e-0abb73137fd8.PNG
[image: capture2]
https://cloud.githubusercontent.com/assets/7411385/11206658/765ef80e-8d4b-11e5-846d-819a42b7c4c5.PNG
[image: capture3]
https://cloud.githubusercontent.com/assets/7411385/11206661/7c1d60fa-8d4b-11e5-9881-0f58e1f1c3fc.PNG
[image: capture2]
https://cloud.githubusercontent.com/assets/7411385/11206682/958ffafc-8d4b-11e5-908f-d4c6d65cffb9.PNG

. It says something like "could not resolve connection identifier". I've
been googling and stackoverflowing for a while but haven't found anything
helpful. Thanks in advance.

—
Reply to this email directly or view it on GitHub
#17 (comment).

[rabyunghwa]

Okay dude thank you very much!

[rabyunghwa]

Hey dude here I am again!

$dbConfig = array(
'dsn' => 'oci:dbname=//localhost:1521/clubsocialapp',
'user' => 'SYSTEM',
'pass' => '123456',
'options' => array(),
'sql' => "INSERT INTO playlists            (ID,field_name,artist_name,title_name,album_name,user_name) values (track_id.nextval, 'hi', 'hi', 'hi',   'hi', 'ByungHwa')",
'empty_string' => '<no track is currently playing>', );

This time it shows the much dreaded "missing or invalid option" error. Well does this at least show that the dsn has been set up correctly and that the php can connect to the oracle database since it says that it's sending the track into database? I've googled everywhere but still have't found anything helpful. Please help. P.S: I'm still using the PDO_OCI thing instead of the ODBC thing. Thanks in advance.

[rabyunghwa]

BTW dude will you ever consider adding support for Oracle Database or any other Relational Database?

[rabyunghwa]

And here is what the playlists table is like:

[ben-xo]

I notice that your fields are all in caps (ID, FIELD_NAME, etc) but your
query has them in lower case. You have to be consistent, oracle is picky.

On Fri, 20 Nov 2015 at 19:47, ByungHwa Ra notifications@github.com wrote:

And here is what the playlists table is like:

[image: capture]
https://cloud.githubusercontent.com/assets/7411385/11310066/6eb978b8-9002-11e5-8b3c-ad80e9ff4bd1.png

—
Reply to this email directly or view it on GitHub
#17 (comment).

[rabyunghwa]

Hey dude. I actually used lower-cased letters when I was naming the columns but somehow Oracle converted them all into caps. However I changed the fields into caps but still it shows the same error. Geez this is driving me crazy. Anyways does this error mean that I got the dsn right? That the php has connected to the database but it just couldnt run the sql command due to a syntactic error? I changed the log level to debug but still its not showing much more info. I don't even know which part of the sql statement went wrong. So dude what else can I do? Thanks in advance.

[rabyunghwa]

So dude what do you think causes this error? Is it still the dsn or the sql statement?

[ben-xo]

Look at this point all I'm doing is googling the error codes from your
screen shot, and seeing what comes up. That's the best I can do to help.
Sorry!

On Sat, 21 Nov 2015 at 16:04, ByungHwa Ra notifications@github.com wrote:

So dude what do you think causes this error? Is it still the dsn or the
sql statement?

—
Reply to this email directly or view it on GitHub
#17 (comment).

[rabyunghwa]

Okay dude! Well the main reason why I want to use Oracle DB instead of MySQL is its support of DB Change Notification Mechanism. Like when a new song is inserted into the DB, the server side(Java code) gets notified of the insert event.

[rabyunghwa]

If I can't insert anything into Oracle DB, then the only option left seems to be periodically polling data from MySQL DB to see if there is any new inserted record, which is far less efficient and ideal.

[ben-xo]

@VxJasonxV so i finally figured out that field 70 seems to represent that the row is an update to a previous row that was written… you often see pairs with 70 = 0 when you load the track, then 70 = 1 when you eject or load a new track. they have the same row ID for the same song, but with extra information in the updated packet. When serato is shut down, these are elided to remove the duplication, so mostly you see 70 = 1 in old sessions.

[ben-xo]

only 6 years later