A curated library of Sysex files

[donluca]

EDIT for better visibility:

––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

Sysex Library v4.0

Thanks to @BobanSpasic for bringing up my attention to newer and more complete packs, we have a new version.
This one is expanded mainly with voices from the "Bridge Music Recording Studio" and some extra voice found here and there.

sysexFinal.zip: This is the plain, normal library, meant for Dexed.

sysexFinalHeaderless.zip: Same as above, but this time the files are headerless.

MiniDexed-sysexFinal.zip: Headered version for Minidexed.

MiniDexed-sysexFinalHeaderless.zip: Headerless version for MiniDexed.

––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

Original post
I'm currently working on a "curated" library of Sysex files for people with Minidexed to enjoy.

I've started from this library where I've found a lots of duplicate files/banks and sysex files which don't work at all.

Right now I've done a major cleanup and I still need to fix some of the banks, but I'm at a good point and I just need to figure out how to merge sysex files which have just a single voice in them and I'm waiting for BobanSpasic to finish his work on his MDX_Tool to fix the remaining files.

Meanwhile I've added banks from other FM Synths thanks to the fantastic work of 4OPtoDX7 which added converted banks from 4-OP and 2-OP Yamaha synths for the DX7.
Those patches won't sound obviously 100% faithful to the originals, but they'll still produce sounds which, well, it's kind of the whole point of a synthesizer.

I've also found lots of other new packs from dxsysex which weren't part of the collection I've started working with, so in the future I'll probably add those as well.

Right now there are ~2300 unique sysex banks in this Library and they have all been named in a way that they are ordered by pack (ex: all the original DX7 patches have their file names start with "DX7-" so they are grouped near each other).

As soon as Boban finishes his work on MDX_Tool I'll fix the remaining patches and publish a first revision of the collection.

[BobanSpasic]

So, I've attached the repaired collection from your previous post. I'll continue my work on MDX_Tool after the school vacancy (after 11. April).
I've repaired the files by using MiniDexedCC (does not check if the checksum is OK). For repairing some of the files, I've used a hex-editor, etc.
Dexed v1_0 (Black Winny)_repaired.zip

[donluca]

Thank you so much!

I already did use MDX_Tool to repair that collection, but I've found some other banks you probably fixed manually and I've added them so now that library is "clean".

In the following days I hope I'll come up with a v1.0 of my library to share here.

[donluca]

Here's the first revision of the library
sysexNew.zip

A few notes:

• Banks which were made explicitly for the DX7II/DX7S have been omitted because they rely on features that Dexed doesn't emulate yet.
• In the original collection there was a huge folder called "!Instruments" which had all the patches organized by instrument type. Those have been omitted here because they're redundant, eventually I might do a separate library organized that way.
• Original Yamaha patches have been given priority, starting with the DX7 own original patches for "authenticity" reasons. This way, when you start Minidexed you have the original DX7 banks loaded, followed by the Voice Cards and then all the other Yamaha synths.
• All files have been tested and fixed with Boban's MDX_Tool and they should all work alright.
• Other files have been manually fixed by Boban himself.

This is a first release so there might be patches which still don't work correctly and unfortunately we won't know until someone goes through all of them (which is kind of insane since the collection is made currently of 2249 banks).
As always, feedback is appreciated!

There will probably be another revision once Boban finishes his work on his MDX_Tool which will probably clean up the rest of the broken patches (if there are any which are still broken in there).

Final note: I don't think I'll add any other banks in the future because there are plenty and I'm sure that there's going to be a lot of redundancy in the voices already, so there's really not much point in adding more.
If anything, I'd be down to remove some if people find banks which are really similar and there's no point in keeping both.

Let me know if you also want the "instrument" library, if there's enough interest I might do it at some point.

[BobanSpasic]

01063_cdDX7SK95.syx - an intruder. It is a file from SysEx Manager (not a plain SysEx), and it contains just TX7 function data. Not relevant for a DX7 or for MiniDexed.

[donluca]

Nice catch! I'll remove it ASAP from the collection.

[Kirtai]

Have you seen the https://www.kvraudio.com/forum/viewtopic.php?p=5996126#p5996126 ? It's a deduplicated and fixed set of sysex files.

[donluca]

That's what I've been using since the beginning. If you look at the first link you'll see that it's exactly that set.

[Kirtai]

I'm surprised. The whole point of the set I linked was that it was supposed to be deduplicated and curated.

[BobanSpasic]

I did take a look at this pack from BlackWinny some 3 months ago. It is still full of dupes inside the banks - the voices are repeating, and not just once.
If you use MiniDexedCC's data bank feature to store the voices, a SHA256 hash of every single voice is calculated (voice name is not included in the hash) as a unique ID for voices. This way, you can find the duplicates of the every single voice inside the banks.

[probonopd]

Using a SHA256 is very clever @BobanSpasic but of course it will catch only exact duplicates. I could imagine that there are many same or almost-same sounds with different SHA256 hashes. I was even thinking of going one step further and using KI to sort "soundalikes". What one could do is render samples for each voice using something like https://github.com/bwhitman/learnfm, and then sort the rendered samples using something like https://www.sononym.net/. Then, one could make banks of similar-sounding voices.

Similar to https://soundcloud.com/bwhitman/the-31380-unique-voices-of-the-dx7

I collected as many Yamaha DX7 patch files as I could and deduplicated each voice within each patch file. There were 59,907 SYSEX files, most of which contained a bank of 32 voices each, for a total of 1,853,728 voice sounds. After hashing the non-name content of the voice (the actual parameters of the sound), I found 31,380 unique voices made by musicians over the years for the DX7. Here they all are. Tag your favorite.

[donluca]

New v1.1 revision out.

I've done some more cleaning, found some more duplicates and managed to "fuse" some banks which had lots of empty/repeated voices.
sysexNew-v1.1.zip

[diyelectromusic]

For what it's worth, it took something like ~3.5 mins to load this entire collection from SD card for me - which is pretty much the same time it took to copy the whole lot to the SD card from the PC in the first place.

I'm not sure how manageable this is mind - it took me getting on for a whole minute of turning the rotary encoder to scroll through them all and that was with no time to read any of the names! :)

Kevin

[donluca]

This is the result of my own run.

I think you're onto something, because I'm experiencing the same issue as you do.

It starts off great, it only takes 1.30 seconds to load 100 banks, and then it drops off.

[diyelectromusic]

Ok, I have the explanation from rene - rsta2/circle#357 (reply in thread)

Yes, opening files has to parse the whole directory chain each time, so it isn't directory handling, but file handling in large directories and is an inherent limitation of the circle FatFS.

Kevin

[donluca]

Well... that sucks :/

I wonder if it is a "bug" or if there is some specific reason why it handles files that way.

This also really makes me want to try and code a caching system in MiniDexed which, as I suggested before, would load all the sysex files and produce a single, massive file with all the sysex files concatenated (plus a header with the file original file names so we don't lose those).

Coding the cache process shouldn't be hard, but I think it would be a bit of a pain to write a cache loader.

Making the header's length fixed would definitely help, like 32 characters (6 numbers + underscore + 22 sysex filename + dot + extension) and we could pad with empty spaces the rest which would then get stripped when loaded.

EDIT: thinking about it for a minute... we're trying to solve a problem which is not inherent to MiniDexed itself, but rather of how a function of an external library works, so I don't think we should put more effort than you already did into this.

[diyelectromusic]

I wonder if it is a "bug" or if there is some specific reason why it handles files that way.

Filesystems are hard. Embedded filesystems are harder :)

It isn't a bug, it's a design choice that favours implementation simplicity (although you might not think that seeing the code) over performance and 1000 files in an embedded filesystem is probably not too typical.

EDIT: thinking about it for a minute... we're trying to solve a problem which is not inherent to MiniDexed itself, but rather of how a function of an external library works, so I don't think we should put more effort than you already did into this.

No, I'm done with this one now, but I just wanted to know the why and now I do :)

Kevin

[donluca]

Agreed, today I'll finish my benchmarking, report my findings, make the necessary changes in the library and move on.

[diyelectromusic]

Ok, @donluca how does 14 seconds sound for your 2238 files? I've split them into directories of 100 and it seems a lot faster with this branch: https://github.com/diyelectromusic/MiniDexed/tree/SysExSubDirs

I /think/ it has loaded them all!? They all appear in the menus and it seems to be able to find the patch names... obvs I've not tested them all ;)

Kevin

[diyelectromusic]

All good ideas, but syx is standard, 1024 voices requires pre-processing before saving onto the disk, with headerless support we can probably load most files that exist, and now we can load 2500 voices in under 15 seconds, I really don't see the need to do anything "out of the ordinary" and complicate the code any further?

Kevin

[donluca]

Agreed.
Now that we managed to cut down load times to a way more bearable level, I don't think we're really interested in other alternatives.

[Banana71]

@donluca , @diyelectromusic
you are doing great things 💪

[donluca]

Not in my case, in my private repo I have removed all of the checks :P

Errr, ok - I'll take your word!? The code checks will be taking negligible time tbh.

I see you're still trying to read an entire TVocieBank structure which of course it won't be able to do, but that is still asking the filing system for 4104 bytes. Maybe try one more time with 4096 as the size? But as I say in the latest codebase it won't save anything anyway so it would only be for academic interest...

Oooopsie! I don't know how but I managed to completely ignore this post of yours!

You're absolutely right, I've made the necessary changes to the code and the result is, honestly, quite impressive: 2.080s vs 2.510s!
That's a nice improvement for just stripping the header (which, to the best of my knowledge, has no use in Minidexed).
Almost half a second shaved for 250 sysex files means that for 2500 files that's 5 seconds off!
I think that for my own use I'll go for headerless files to save some extra time.

EDIT: this also means that my theory is probably/partially correct about aligning file sizes to the unit allocation size of the SD card :P

[donluca]

9.859s full library, using headerless files!

I wonder if there's anything which can be done to push this even further.
I can definitely buy a better SD card, for starters.

For reference, here's the exact same library with headers:

Almost 14 seconds!

[diyelectromusic]

Now submitted: #473

[BobanSpasic]

A side quest - MDX_Tool updated - now, it can rescue some voices from incomplete dumps, and it will extract DX7 VMEM from TX7 and DX7II dumps.
Just Windows version released (I am on vacancy in other country, with just a Windows laptop with me).

[donluca]

Right now I'm busy doing benchmarks, but I'll give it a go as soon as I can and provide a macOS binary as well.

EDIT: done.
MDX_Tool for macOS
MDX_Tool.zip

[diyelectromusic]

By the way, anyone else find it annoying that the bank numbers shown on the menus is the bank number from the filename +1? This makes sense for things like MIDI program numbers, but if a user has put a number in the filename, I think the menu should show the same number rather than try to avoid having "bank 0"?

Thoughts?

[donluca]

So do we ignore bank 00000 and just start from 00001? I.e. subtract 1 when reading banks in to get to our internal 0-indexed banks, then add 1 in the UI to show it back to the user?

That's what I proposed in my previous message.

[diyelectromusic]

Sorry, I thought you were just talking about filenames! If we didn't change anything and just started at 00001 then the UI would start at 2 :)

Anyway, update on its way to use 1..indexing in filenames and UI but sticking with 0..indexing internally and MIDI. Of course this will mean that from this release people will have to renumber their existing sysex files by 1... :/

I guess we /could/ make it a config setting... but I'd really rather not. There are already too many!

Kevin

[donluca]

I think I have a clean solution for this:

let the UI show "Bank 0": that bank is the User Bank with the empty User Voice which can be used to make custom pathces/performance.

000000_User_Bank.syx will be always included and will be the first bank (bank 0).

EDIT: on a second thought... that's a terrible solution due to what's been discussed in #472 about default power-on behavior.
It could be prevented with a default performance.ini which makes the user start at bank 1 on first power on, though.
What do you think?

[diyelectromusic]

As MIDI and internal banks count as bank 0, but the MIDI spec says banks should be visible to the user 1 to 128 (or 16384) I'm afraid that having a file as bank zero would be a lot of complications in the code as it would need special cases everywhere.

Kevin

[donluca]

Yeah let's scratch that idea, not a good one.

Let's just follow what the wiki says and put that user bank at the end, so we'll stay consistent with the documentation.

[donluca]

Thanks to the latest version of Boban's MDX_Tool I've managed to discover some more duplicates and do a further cleanup.

I think that this is the definitive version, at least for me, as I don't plan on adding/removing anything else.

There are 2240 banks, obviously I couldn't check every single of them, so please let me know if there's something broken/not working and I'll remove it!

Also bear in mind that from this version onwards, you'll need to have subfolders support ( #473 ) as this not only allows to have a much more manageable and readable collection, but will also massively improve loading times (from almost 5 minutes down to 14 seconds or 9 seconds if you use the headerless version).
EDIT: #473 has been merged into main! Make sure to use latest version of MiniDexed to use this collection!

I'll need to make a PR in order to enable direct headerless loading, so stay tuned!

Normal library v1.2:
sysexNewest.zip

Headerless library v1.2:
sysexNewestHeaderless.zip

[BobanSpasic]

I have some 42k unique voices in my collection (which isn't organized in any way, and contains no original Yamaha voices). After you publish your collection, I could upload the difference.

[donluca]

That would be really interesting, maybe we could create an "Extra" folder of voices which comes from your collection which are not included in this one!

[Banana71]

I won't find time for the next few days, so I'm sending the work I've started. Note the log file.
Tim_Garrett in process.zip

[donluca]

I can handle the bulk of the TX* patches in one go if you want and at that point there's no much left.

Would it be ok for you if I take care of it tomorrow? With that one we're basically done and can upload the new revision.

[Banana71]

Yes, that would be great, I can't keep up with your pace at the moment.
When I have time I will sort and upload all the sounds I made for the miniDexed performances. A few banks come together.

[probonopd]

Now, with that many nice voices, I desperately am looking for an easy way to switch Unison mode on. So that I don't have to change the voice on multiple TGs just for detuning and stereo shifting :)

[Banana71]

Speaking of the Sound Library, the best DX7 patches I know of are on my PLG150-DX PlugIn Board for my Yamaha Motif ES. There are around 900 DX7 patches, some of which are of very high quality. Some classics from the old DX7 series are represented, but also completely "new" (anno 2003) patches.
I haven't found a way to get these sounds off the board yet (MIDI Dump, Save on USB....) Have any of you found these patches on the internet? Or know how to get them from the plug-in board? Help....!

[donluca]

I was afraid this kind of topic would have been brought up sooner or later, so I think it's best to talk about it here and now:

Some of the patches of this library were commercial and it seems like the authors are still around and, despite not being available commercially anymore, are not too happy to see them floating around.

I believe that some have already been removed from this library by its original author, but I still see some (ie: Bobby Blues) that are included and I don't want to bring legal issue on this project, hence why I originally proposed to host the library on my own website, so that eventually they'd come after me and not MiniDexed.

Those "new" patches on that PLG150-DX board might even be riskier to include in the library.
Besides, I've done some searching and it seems that patches stored in those cards are in a custom format and would need to be converted to sysex somehow – a feat no one seems to have been able to do so far.

[Banana71]

We must never jeopardize this great project and violate copyright.

I can send individual patches to my PC via midi bulk and receive them as sysex. With the Hexviewer I can only see a few similarities. The file is 3 times larger than a Dexed single voice bulk sysex.

[BobanSpasic]

It was boring yesterday... so... https://github.com/BobanSpasic/MDX_BankManager
It works just with non-corrupted files.

[BobanSpasic]

@donluca could you take a look at the latest release of the MDX_Tool?
Also, there is a text files with Darwin compiler switches in the source files, as generated by the IDE. You can use it to make a Bash script for compiling your MacOS executable.

About the tool: I would go with -w parameter to scan every file in the collection. The option will add the file extension .nl to the files containing null-bytes in the name. Most of these can't be repaired automatically, but I would take time and save manually what it can be saved.
After that, you can go with the parameter -y through the collection. This will add the file-extension .corr to the files where the parameters are outside the min/max limits. The option -z could normalize such files, e.g. if the maximum value for a parameter is 99, but the sysex file contains 127, the normalize function will write 99 into the file. You can skip the scanning with -y, and go straight to -z (normalization).
Anyway, I would do a backup before normalization, because I didn't tested enough files to see if the normalization works as supposed.

After all the banks are corrected, I would go for -x (--xsplit) option. This one saves the banks as single voices (VCED, still usable even if it remains in this form), where the file name is the SHA2-256 hash of the parameters (excl. voice name). Searching for duplicates is searching for same file names.
Compiling voices back to banks can be done by -j (--join) parameter. It takes a directory with VCEDs as input. If this directory contains a file voices.lst - the voices will be organized in the bank in the same order as in the list. Voices.lst is a key=value list, where key is the hash, and value is the voice name. Hash is used for organizing the voices in the bank, voice name is just for the info in the list. This file is generated at splitting the bank into voices with -x option.

[donluca]

Thanks, it worked!

Tomorrow I'll give it a go and see if it can find anything off in the library.

[donluca]

Thanks to your tool I've found an astounding amount of patches which are completely broken... and by completely broken I mean that whoever did the sysex dumping made a mess.

The bigger offenders are Aminet, Big_Mamma, Jeff Saxe and Larry Stanos.

It's going to be hell trying hunt those down, because some voices are legit and just have some parameters "out of spec", while others are completely corrupted (ie: bad sysex dump).

I have found some more patterns and I'll see if I can clear those.

[BobanSpasic]

In my collection, I've found 168 banks with "null voices" (voice name contains byte $00 - that's why I call them "null voices"), and another couple of hundreds with parameters outside the limits.
Just upload your broken banks for me (the "null voice" banks), and I'll repair everything what's possible. Probably we have the same collection of broken banks from the internet.
The banks with max values 127 instead of 99 - that is probably a result of some poorly written voice editor software.

[donluca]

Thank you so much Boban, with the newest version of your tool I've been able to find, hopefully, almost all "broken" voices (voices which have garbage data) and through some regexp I should have been able to take care of all of the others.

I've then fixed all the voices which had incorrect values and I think it's now time to release, hopefully, the final version.

[donluca]

Sysex Library v2.0

Thanks to all of the involved, we've managed to hopefully have a fully working library with 1023 banks made of unique voices, all working.

There are several versions:

sysexFinal.zip
This is the plain, normal library, meant for Dexed.

sysexFinalHeaderless.zip
Same as above, but this time the files are headerless.

MiniDexed-sysexFinal.zip
Headered version for Minidexed.

MiniDexed-sysexFinalHeaderless.zip
Headerless version for MiniDexed.

Again, huge thanks to all of the involved, especially Boban and Banana71.

[donluca]

Not that I'm aware of.
You get slightly smaller file size and faster loading, but you have to modify the sysexloader code to fully benefit from it and, AFAIK, there's no proper way of doing this, because you'd have to first check for the file size, but that check alone on 1000+ files would add a big overhead and make the use of headerless files useless.

I just have a private repo where I have removed all kind of checks and made MiniDexed expect to always load headerless files, this way the boot time is less than 5 seconds.
Otherwise I think that it is slightly longer, around 8 seconds, if anyone wants to give it a try.

[probonopd]

OK, for me personally the 3 extra seconds are not worth the extra complexity at the moment. Thanks!

[BobanSpasic]

Thank you very much!
Excellent job done!!

I've still found a couple of impostors...
Corrupted voices in banks:
Big_Mamma\DIR03\BIGMAMMA115.syx
Jeff_Saxe\JEFFSAXE003.syx
Jezreel\JINCONI01.syx
Larry_Stanos\LSTANOS002.syx
Larry_Stanos\LSTANOS003.syx
Larry_Stanos\LSTANOS008.syx
Larry_Stanos\LSTANOS009.syx
LiVeMuSiC_Hollywood_Collection\LM_VARIOUS.syx

[diyelectromusic]

As I've said many times already in this discussion already, the main overheads are reading from the SD card itself not the code performing the checking, so I strongly advise against removing all checks. As I said here (#456 (reply in thread)) we shouldn't mix up removing all checks for "performance" and supporting headerless files - the two are different things and i would argue that accepting headerless files means there ought to be more checking going on to prevent MiniDexed from trying to load in any old rubbish,... (there might be a slight performance gain if it was a genuine like for like situation as in principle a headerless file would be a single sector/block on the flash disk, but I'm not convinced it is a significant problem reading two blocks compared to the overheads of parsing the directory structures).

This does not "make the use of headerless files useless" as the support is there for wider compatibility with SysEx voices from the internet, not performance.

This is why the loading code's support for headerless files is disabled by default - it is "less robust" for non-advanced users, but is there as an option if the user is willing to accept they might break something.

If three seconds is important, I'd suggest buying a faster SD card ;)

Kevin

[donluca]

Thank you very much! Excellent job done!!

I've still found a couple of impostors... [cut]

Damn it! 🤣
I was sure I managed to find all of them and fix them!

I'll have to go through those banks again... I guess version 2.1 will be out in the following days.

@diyelectromusic yes I agree, I just took some shortcuts for my own personal use which are not meant for "public use" ;)

[donluca]

Sysex Library v2.1

sysexFinal.zip
This is the plain, normal library, meant for Dexed.

sysexFinalHeaderless.zip
Same as above, but this time the files are headerless.

MiniDexed-sysexFinal.zip
Headered version for Minidexed.

MiniDexed-sysexFinalHeaderless.zip
Headerless version for MiniDexed.

[BobanSpasic]

@BobanSpasic do you have code to convert from/to a text representation for voice data and/or cartridge/bank .syx files (on the command line)?

Use the code from MiniDexed that reads and saves the hex-sequence in performance.ini files.
dxconvert (mentioned above) can display a text info, but there is no text>binary conversion, and the format above wouldn't be easy to parse,

[probonopd]

...but it'd be human readable.
And dxconvert.py can convert it back to .syx.

[donluca]

@BobanSpasic I think that the txt format of DXConvert, while being a pain to parse, would still be nice and easy to edit "by hand", so that people can eventually edit/compare the parameters if needed.

Also, I'd like to raise the question whether having the .syx files on github in a human readable format would really matter.

You technically want such a thing so that if someone has to edit (for whatever reason) a certain patch, he doesn't have to upload the whole thing, but just a small diff.

But here, once unpacked, each voice are a handful of bytes and even a complete bank is 4096 bytes (or slightly more if headered).
Is it really worth it instead of just uploading either the unpacked banks or the whole bank?

[probonopd]

Git and GitHub and the tools and methods used in collaborative software development (e.g., diff, pull requests,...) are made to handle source code (text) files (readable by humans) very efficiently.

So we need to find/define a way to express voice data as "source code" (text files rather than binary blobs).
Then we can use all the power of collaborative software development tools.

[donluca]

I know, but in this context we're using Git... not strictly in the way it was intended, if you get what I mean.

This is more like an indexed database to keep track of what's in the library and eventually correct errors/expand the library, it's not really about people editing the voices (if one voice needs to be edited, it's because it's corrupted, and if it's corrupted than the conversion to txt will probably fail or give unpredictable and potentially misleading results), but rather having a place where people can contribute to add/change banks/voices to the library in an easy and efficient way.

Just some food for thought, as I said previously I still like the direction you'd like to take this to, but maybe it's just adding unnecessary complexity and asking for trouble :P

[donluca]

Sysex Library v3.0

• Added extra instruments that were in Whinny's collection under a separate "Instruments" folder. I thought those were all duplicates, but it turned out that there were several new voices!
• Fixed a mistake where, for some reason, one of my scripts duplicated the voices in v2.1
• This is the real real REAL final release, unless issues/mistakes are found :P

sysexFinal.zip: This is the plain, normal library, meant for Dexed.

sysexFinalHeaderless.zip: Same as above, but this time the files are headerless.

MiniDexed-sysexFinal.zip: Headered version for Minidexed.

MiniDexed-sysexFinalHeaderless.zip: Headerless version for MiniDexed.

[probonopd]

voice.zip Version for Minidexed in a format suitable for storing in git and for making human-understandable diffs. Converted using dxconvert.py from DXconvert-3.1.7.zip.

[donluca]

Oh, nice.
I already knew about DXconvert and it was the first place I looked into, but on the website it didn't say that it was capable of doing this. Good to know!

[probonopd]

Do you want to put those into a GitHub repo? Should be its own repo, not in MiniDexed.
Then we could try to automatically generate the output format(s) using GitHub Actions.
I might send some pull requests if I can get it to work.

[donluca]

As I said, I'd prefer someone else to take over this library and become its maintainer.
As long as all the people involved are mentioned somewhere I'm 100% fine with it.

[donluca]

I've published the tools I've made to make this library: https://github.com/donluca/DX7_Sysex_Tools

They are bad and no one should ever use them, but since I've put quite a lot of time into them, I decided to share them.
Maybe @BobanSpasic can get some inspiration from them to add more features to MDX_Tools.

[BobanSpasic]

Thanks for your hard work. I am using now your collection as the basis for my collection. Yesterday I have sorted out 35k voices difference between your and my collection. There are a couple of things that I do not understand, like - I have downloaded 3-4 different Black Winnie collections from the internet (Yan Anselmo's copy being the biggest one), and there alone are 25k voices more than in the collection you published (no duplicates at all).
The rest 10k voices are from GeminiSolstice (Chick Korea's backup, Deckart's backup etc), SynprezFM, DX7_Lib (program distributed with huge collection of voices).
Now, all that is stored as single voices. I'll need to compile banks before I upload the collection somewhere, and that is still a lot of work to do...

[BobanSpasic]

btw: https://github.com/jphaenlin/DX7-patches - another ready-to-use collection

[donluca]

Mmmh... I'm afraid that people have been using Winnie's name to make their own collections, which means that there are several libraries floating around the internet with the same name, but different content.

I completely forgot about SynprezFM, I could have added them because I was aware of them.

[donluca]

Ok @BobanSpasic , curiosity had the best of me.

I've downloaded the Anselmo collection and I'm going through it as we speak.

I'll report back and eventually make a new update, although it feels really strange that there are so many missing voices from the collection I've made.

[donluca]

I know what happened and the whole story of those packs.

In the beginning, someone named "Solstice/GeminiSolstice" shared a pack named "Bridge Music Recording Studio" which was a very well curated pack with lots of banks from several artists plus official (and non-official) Voice Cards for the DX7 and many more which the description is very vague and doesn't tell where they come from.

From this collection then people started expanded it and here comes BlackWinnie who did his own collection by adding extra voices (and removing others? Possibly).

From there, it became quite chaotic as people started making their own collections by keeping the name of BlackWinnie so now we have a ton of different collections which are very very similar, except that some have some extra packs in them.

The Anselmo's collection was almost identical to the one we had, except that it had also the original "Bridge Music Recording Studio" collection where I found several new banks and voices.

Needless to say, I've found an abundance of corrupted banks and voices, so, again, I hope I've managed to clean it up well.

I've posted the new version down below.

[donluca]

Sysex Library v4.0

Thanks to @BobanSpasic for bringing up my attention to newer and more complete packs, we have a new version.
This one is expanded mainly with voices from the "Bridge Music Recording Studio" and some extra voice found here and there.

sysexFinal.zip: This is the plain, normal library, meant for Dexed.

sysexFinalHeaderless.zip: Same as above, but this time the files are headerless.

MiniDexed-sysexFinal.zip: Headered version for Minidexed.

MiniDexed-sysexFinalHeaderless.zip: Headerless version for MiniDexed.

[BobanSpasic]

MDX_Tool update. MDX_Collect added. Please also check the Wiki: https://github.com/BobanSpasic/MDX_Tool/wiki
I'll add more to the Wiki, inclusive repairing the files with a Hex editor.

[donluca]

Excellent writeup!

Believe it or not, I've been looking for such documentation forever and it's incredibly nice to have it all in one place and so well explained.

Thanks for mentioning me, I'll share your project and your wiki on my social media outlets.

EDIT: I have one request which I always keep forgetting to ask for MDX_Tool: can you add an option so that the normalized or repaired files are just overwritten instead of having new xxxxx_DX7_repaired.syx, etc. files?

[BobanSpasic]

EDIT: I have one request which I always keep forgetting to ask for MDX_Tool: can you add an option so that the normalized or repaired files are just overwritten instead of having new xxxxx_DX7_repaired.syx, etc. files?

It will be done, but first I need to fix the problem with hashing the files. This one bothers me a lot.
Btw. be sure to check the Wiki from time to time. I am adding pages. The 2nd page is already there. If anyone finds mistakes in the Wiki, please report. I am not a native English-speaker, neither I've learned English in school. I've got it from the TV, and from the PC.
My native language is Romanian, 2nd language Serbo-Croatian, 3rd language German and 10 years of learning Russian in the school (but I can't even bring together a single simple sentence in Russian...)

[BobanSpasic]

I believe, the MDX_Tool got into the final form. I've implemented pretty much everything I've planned and also implemented the suggested features.
I am working on MDX_Tool for 4OP synths now. It is much more complicated :)

@donluca
Please check the TX802 banks - 2 files are corrupted (too short),

[donluca]

Confirmed, thanks. I have no idea how that could have happened since the Yamaha banks are left untouched.

When I get back from vacation I'll investigate and fix it.

About 4OP (and 2OP) FM synths, you might want to check out this: http://home.kpn.nl/m.tarenskeen/download/sysex/DXconvert/dxconvert.html

There could be interesting information buried inside the code (it's Python).

[BobanSpasic]

I know Martin's site and programs very well, and I use them too, but his code is hard to follow (not impossible, just hard).
What I need, to sort a 4OP collection, is to split VMEM bank into separate voices, and that is not easy if you do not want to lose data.
A single 4OP voice can be a simple VCED, or VCED + ACED, or VCED + ACED + ACED2, or VCED + ACED + ACED2 + ACED3.
It is not so simple, because the VMEM is the same for all the forms of single voices, and you need to find out into which form of single voice you need to convert the data from VMEM.
As for 2OP converter - it is very limited. I own two 2OP synths (PSS-580 and PSS-780). DX7 can work just with sine waves, and 2OP series have 4 different waves for oscillators. Neither the DX11 (or latter models) have all these 4 waveforms (sine, half-sine, squared-sine, half-squared-sine).
The same problem as translating the 4OP voices to DX7 - DX11 and later models, have 8 waveforms for oscillators, not just sine wave.
The success rate of the conversion depends on, if the 4OP/2OP voice was from sine wave, or some other waveform.

[Andy2No]

What's the procedure for copying those into the /sysex/voice directory?

In MiniDexed-sysexFinalHeaderless.zip I see folders inside folders, each containing a few .syx files. Do we have to take the files out of each sub folder and put them into /sysex/voice without having folders inside it, or can we just copy the whole lot in there as it is, still inside sub folders?

Is the idea to put all of it in, or is that too much for MiniDexed to cope with?

[donluca]

Copy paste the folders into /sysex/voice and you're good to go!

[Andy2No]

@donluca Thanks. It tries to process them, but it just gives a long string of "invalid size or format" messages, one for each file, then says "syxfile: 0 banks loaded. Highest Bank loaded: #1".

That was with MiniDexed-sysexFinalHeaderless.zip Apr 24, 2023. I was asked if I wanted to copy the contents without encrypytion, but the alternative seemed to be not copying them at all, so I assume that's normal. I seem to be fundamentally failing to grasp something... I'll try the version with headers next.

[Andy2No]

@donluca "MiniDexed-sysexFinal.zip: Headered version for Minidexed." seems to have worked - it now says 1302 banks loaded.

[donluca]

I don't remember if you have to use the modified version of MiniDexed from my own fork to use Headerless files, but the ones with headers are the safe bet.
Maybe they're going to take a little bit more time to load, but it should not be game breaking.

[Andy2No]

Thanks, @donluca

[Kirtai]

This set might be useful if someone can get them to work.

[donluca]

@BobanSpasic I see, thanks as always for all your help.

I'm in the middle of something with two other projects, I'll try my best at adding those Volca FM patches to the collection as soon as I have some time on my hands.
Meanwhile, if someone can share the patches of the Volca FM2 would be great, so I won't have to update the collection again in the future.

[cjfp]

@donluca These are the FM patches:
https://www.reddit.com/r/volcas/comments/mdw6p7/heres_a_sysex_of_the_default_volca_fm_patches/
https://drive.google.com/file/d/1HxMkFVNKmLmjFfHJXDhnTm5ooT8onVAZ/view?usp=sharing

These are the FM2 patches:
https://www.reddit.com/r/volcas/comments/wyh8uo/volca_fm2_factory_patches/
https://drive.google.com/file/d/1e5YuDBOV0kLXeAlZ_IwZeaqLhJ7hynkL/view

I don't know how those differ from the 3 OscillatorSync patch banks linked above:
https://github.com/oscillatorsink/synth-patches/tree/master/Korg%20Volca%20FM

[donluca]

Thank you so much for your work.

Unfortunately I'm not sure when I'll be able to refresh the collection, I wish someone would take over this and continue updating this curated library.

I'll still add it to my to-do list.

[cjfp]

@donluca I don't have time to do it either, but there's no rush I think, you were working for free and the DX7 is 40 years old! I looked at your scripts and they were easy to understand and well documented. So thanks for leaving things in a good state.

[donluca]

For the love of god, don't use them.
They are terrible and I uploaded them just to document what I've been doing and to vent my frustrations.
@BobanSpasic made way better tools which are available at https://github.com/BobanSpasic/MiniDexedLibrarian

Please use those.

[Onosendaifr]

Is the first page edited with the last version of the archive ? Thanks a lot for this !

[donluca]

Yes, I've kept the first post always updated with the latest version.