Replacing sounds?

[geist22]

Edit for anyone reading this in the future:
Thanks to the amazing work of @Moburma and @mefistotelis, it is now possible for users to create their own, custom SOUND.DAT files. Instructions can be found on the wiki.

Glad there's a discussion section here now, I was hoping for this for a while but didn't know where to put it, as some topics don't really fit under issues. Guess I'll be the first to open a thread here 🙂

Anyway, since the recent discussion about the japanese version I've been thinking about replacing some of the ingame sounds. Specifically, using the newer variants like german/japanese version of Electron Mace, out of energy, possibly mail/mech, because they are newer and can be considered a more definitive version, and for nostalgia's sake.

But I have no idea how to do it. As I understand it, all sounds are in SOUND.DAT, some proprietary container format from EA. Of course it wouldn't be as easy as just using another version's SOUND.DAT, as the german version has german audio, and the japanese version misses the billboard ads. So I guess you'd have to unpack, replace and re-pack the entire thing.
I see one of you guys already did the work and made a tool that can extract SOUND.DAT, though I can't seem to find it anymore right now. I also have not found a way to re-pack the files back into a working SOUND.DAT.

Is it possible to do this, and if so, can someone provide step by step instructions on how to do it? I'm using the english version as a base.

[mefistotelis]

Currently, we don't have any tools ready for easy gfx and sfx modifications. But we are moving towards that.

Here's the important part I wrote about it in swars-re-helpers discussion:

What I want is not an editor, but sound bank builder.
Like what I did with keeperfx. There's a lot of WAVs and filelist.txt which contains their names in specific order. Then a sndbanker tool builds a sound bank out of these.

So any modifications do not require editing binary file, only adding new WAVs and modifying the filelist.
The sound bank is auto-generated.
Any downsampled versions of the WAV files should also be auto-generated (I did that for DK too, but I don't think the mechanism is currently used).

Bullfrog has a generic sound bank support in their bfsound library, but that was evolving over time.
And in SW, they actually disabled that generic support to implement game-specific sound banks - the DisableLoadSounds is set, so that bfsound library does not handle the bank, and setup_heap_manager loads them instead.

Rules for adding sounds to swars-sfx repo:

• Let's place the files in sound folder
• we should have sub-folders for types of sounds. like weapon sounds, people sounds screams, weapon names, environmental sounds, vehicles, etc.
• for languages, I guess only specific sub-folders will have to be per language, the rest is always the same. so train speeches should be separated from the rest of train sounds.
• Any files without effects added or in higher fidelity than the ones used in game, should be placed in sound_design folder.

In general, what we need:

• WAVs and file lists for sound banks - WIP by @Moburma
• SW sndbanker tool, either in C or in Python - we will see
• Scripts to automatically build a new release of sound banks when change is made

[geist22]

I see, so it's currently not possible, at least for a layman like myself. But I'm glad to read that there are plans to make it possible in the future.

As the question is technically answered, this can be closed. We can edit the posts later when the time has come.

[Moburma]

At the moment I propose the following for structure, but it might be a bit over the top. Let me know what you think and if it should have fewer categories:

AdvertsEng - The billboard adverts. There would be one of these for each language
ArkanoidMinigame - Self explanatory, I was actually wrong, the early versions DO have different filenames so they can mix
CombatSounds - screams, ricochets, etc
EmailEng - English samples for new email and starting mission in menus. Other languages follow
Explosions - Self explanatory
InGame - Anything else in-game (i.e. not menus), doors opening and closing, the ambience loop, etc
InGameSpeechEng - The English "level" sounds, police speech, also some UI sounds like "selected"
Menus - Menu sounds, drawing, confirm, cancel, mods screens
TrainAnnEng - Train announcements in English
Vehicles - Sounds for cars, Tanks, train movement
Weapons - Weapon firing sound effects. Also things like medkit use
WeaponSelectEng - English version of the weapon UI speech, "Uzi", "minigun", "plasma lance" etc

The only thing we really MUST segregate are the different language files, they have the same filename so can't coexist. Each category with "Eng" above would of course have a corresponding "Fre", "Ger" etc folder. They were actually very good at renaming new versions of files, I don't think there are file conflicts between new and old versions of sounds, they are uniquely named.
There will also be a folder for the pre-alpha sounds (as they don't fit 1:1 with the final game, but probably could do), and also "spot effects", which is the cut spot effects that only work in the beta demo (weird ambient wooshy sounds that would randomly play in the background).

I can then just stick a relevent filelist.txt in the root, so you would have like FilelistEng.txt, FilelistGer.txt with each version's sounds set up as the original file. Then we can make new versions with e.g. all the new sounds added to the original english game etc.

[Moburma]

I think I can, problem is I can't get the already created Windows release to run at all! I just get the error "/: Permission denied". I wanted to see how it would behave with the files as-is before any changes.

I did some research into the file format and wrote it up before.

Cliff notes: We only need to change the termination header on the samples and put in support for the second low quality bank and its header info at the end of the file, the format is 95% identical:

Sound.dat

Last four bytes of file are pointer to start of wav file table information - same as DK
Four bytes after pointer location (five bytes including) is the number of sound banks. Very few files have more than one, the Japanese sound.dat does for some reason - same as DK
12 bytes from pointer location is I believe the number of low quality sound banks. NOT in DK (or at least always 0 there as no low quality sounds)
50 bytes from pointer location is start of the filename tables location in file - same as DK
32 bytes before filenames is a "box" of bytes with FF. These also come after the data - same as DK (with exception for low quality data)
Contents of first part of the "box":
4 bytes - location for start of the high quality filename table location in file - same as DK
4 bytes - offset for start of samples in this bank (first one is always zero - i.e. start of file, low quality are always offset later on) - same as DK I imagine (doesn't have a low quality bank to prove it)
4 bytes - length of this file information table in bytes
4 bytes - length in bytes of all samples in this bank
FF bytes then continue. In DK these go right to the end of the "box", in SW there is another sub box that contains the low quality sound file information
48 bytes of FF from length of samples in this bank
Low quality sound bank sub box. Same set of 16 bytes as above, but now the offset will be someway through the file where the low quality sounds are concated, and not zero. This is all different from DK
32 more bytes of FF until the end of the "box"
16 bytes of 00 and then the LENGTH (this will be the same as the start address for bank one, but it's not that) of the first bank (high quality) samples is stored here again for some reason (4 bytes) - Same as DK
The length of the file information table for the high quality bank immediately follow (4 bytes) - same as DK
24 bytes of 00 later, The length of the low quality sound samples files can be found (4 bytes) and again, the length of the file information table for this bank immediately follows (another 4 bytes). Not in DK (or at least not used)
16 more bytes of 00 and we hit the final pointer 4 bytes back at the top of this list

Japanese version is different. It has TWO set of sounds banks (2x high, 2x low). It's the same as above, the first "sub box" of data is 32 bytes of FF in again, but the next (bank one low quality )one is now 48 FFs instead of 32. Then the FF divider between banks is 64 bytes from bank one low quality to bank two high quality. Then it's 48 FF again for bank two hi to low, and 32 bytes FF to finish. It also oddly only seems to then include the repeated length data for bank one, not bank two.

Sample table entries

Each entry is 32 bytes long. The first one seem to be a dummy entry, BUT it has a 4 byte value 26 bytes in it I'm not sure of yet. This value is in the DK one too, so should hopefully be known there
Each entry starts from the first letter of the filename -same as DK
Filenames are 18 bytes long (but never use this much space as they are 8.3 compliant) - same as dk
Immediately after is the start position of this sound sample in the file, 4 bytes. Note this is OFFSET versus the stated offset value for this sound bank set in the box data above. For first bank this is always 0, but for later banks and low quality bank it is this position + offset - same as DK but DK doesn't need to know offset as always 0
4 bytes - 00 padding - same as DK
4 bytes - length of this sound sample in bytes - same as DK
2 bytes end header - this is always 5A 00. This is the main difference between it and DK, DK does not seem to have a header here. The first "dummy" entry does not have this header

[mefistotelis]

This should help you:
34be276

[Moburma]

Not had time to do much on it today due to having to do damned family man stuff, but hopefully I can tomorrow. Ripping the rest of the WAVs now, and realised something really dumb I hadn't noticed - the French version actually already had the mech and new sound effects in it all along. I thought it was only the later German release that had them, but they were on the release CD from day one...
Strangely the Italian sounds don't have these, not done all the rest yet.
Edit: Yep that's weird, ONLY the French version has those new sounds on the disc, all the other languages are the same as the English one.

[Moburma]

Ok, banged my head against the wall with this for ages. The sndbanker tool refused to output anything from my source files. It worked fine for keeperfx, but was just giving a weird "./: Permission denied" error with the SW files. I even got a linux box running to test... same there.
Then I checked the text encoding. I somehow saved the main .txt file as UTF-16 LE. All the keeperFX ones were UTF-8 BOM. I think you can work out the fix needed...

After all that sndbanker already produces SOUND.DAT files that work perfectly with Syndicate Wars. They're not 1:1 identical as they don't have the same terminator on the file table entries, and they also don't have the second low quality samples entry at the end. But the game doesn't care about the former, and isn't using the low quality samples anyway. So, do we care?
Here is a SOUND.DAT of the original English sounds with the new German additions (new electron mace and email sounds, and the mech sound added) I created using sndbanker.

[geist22]

Outstanding work! That SOUND.DAT is exactly what I was looking for.

So with the swars-sfx repository and the sndbanker tool, users could build their own, custom SOUND.DAT files now?

[Moburma]

Outstanding work! That SOUND.DAT is exactly what I was looking for.

So with the swars-sfx repository and the sndbanker tool, users could build their own, custom SOUND.DAT files now?

Yes. There are some small differences though, I roughly copied the KFX format, but the .txt files there have a header at the start of the file. Currently you MUST add a dummy line at the top of the file as the tool expects it and the resulting .dat will be one off and you'll get weird sounds playing for the wrong thing. Also the way the game works is it doesn't care about the names of the files, it works by which "slot" each sound is in. It will always only use slot X for say the minigun fire sound, so you can stick anything in that slot and it will play when you fire a minigun. That's how they could quickly change between sound versions, you just replace what is in the slot (Theme Hospital also works like this, they made a big mess of many of the announcement sounds there by putting them in the wrong slots). So if something's missing from your version that is in another version, just compare the two text files in a spreadsheet and it's obvious what needs to go where (like the mech sound has a specific slot it needs to go in that is normally empty). The text file is used to order the files so is the crucial part of the process. You can also do stuff like add in the missing police speech from the beta demo (LEVEL008 and LEVEL009.wav) now.

At the moment the file format is not technically 100% correct for Syndicate Wars, but it doesn't seem to actually matter, so it's kind of an open question if it's worth reinventing the wheel to "fix" this or not - probably not. If we do decide to just keep it as-is then I need to redo the text files in proper utf-8 format and with a header at the start so they work properly with the existing DK tool.

[mefistotelis]

They're not 1:1 identical as they don't have the same terminator on the file table entries, and they also don't have the second low quality samples entry at the end. But the game doesn't care about the former, and isn't using the low quality samples anyway. So, do we care?

When making the sndbanker, I was including support of all DK features. We should do the same for SW.
The tool should be modified to properly support SW features.

Lower fidelity sounds unused - then we can leave this bank empty. But an entry for it should still exist.

I don't really see a situation where we would use the low spec sounds, but we are keeping low res mode alive... I don't know. For now, we can leave that as empty - I'm pretty sure OpenAL will never report it only supports low spec sounds. It could be a command line switch, but who would deliberately use a switch for worse sound?

[Moburma]

Ok, I've been a good boy and actually done my homework properly:
https://github.com/Moburma/sndbanker
Not much is different, but it now sets the "sfxid" to always be 90 as it is in SWars files, and fills in the low quality sounds area of the footer with a copy of the high quality values, so if it did somehow load it it would go to valid data.
Because of the truncation of samples code you can't get exact 1:1 of original files still (without the low quality data, obviously), but it looks identical otherwise, and works fine in-game.
The Japanese version of the game shows this isn't all that is possible with the file limit. I'll think about adding this support. Basically it seems that with more than 230 entries needs a new sound bank. However a) these seem to be mostly duplicates and old files and more importantly b) I don't think the game actually even uses them at all.
So it would be more correct, but also useless.

EDIT: Just tested with Japanese input files, it works fine as-is even though technically wrong with too many entries in one bank.

[Moburma]

Some other musings on the sound banks:
Was actually a good exercise listing all the files from each version as it makes clear some patterns in them and highlights obvious cut sounds (like clearly we are still missing some "level" sounds they made). Also some of the final menu sounds are marked "Z" for zealot. This suggests that at some point they had different sounds for zealot and eurocorp menus, then gave up on that idea.
Also the pickup weapon sound has a note that originally it was a weapon CHANGE sound, so I guess at one point it would make a kind of weapon cocking noise when you changed guns, that probably got dropped once they brought in the "UZI, MINIGUN" etc samples.
The cut "spot effects" are duplicated many times in the beta demo file. They probably did this to "block out" the space in the file to reserve for these, so there would be lots of these random sounds to choose from. But then they gave up on the idea entirely, so you have this massive empty space there in the final files. The beta demo also has loads of common sounds blatantly cut out to stop spoilers (you can't use anything past the laser in the demo). Kind of a shame as a lot of these were likely earlier sound versions at this point.

[mefistotelis]

Congratulations on your promotion, executive @Moburma - you're now an admin of swfans.

You can now move your sndbanker repo to the organization.
(or we can just clone it, doesn't make much difference)

What next:

• prepare automatic builds for sndbanker
• make a release of sndbanker
• Use the release to add automatic building of sound banks

[Moburma]

@geist22 do you definitely have the executable in the sound folder?
It should look like this:

Also are you changing directory to that location first as well (pro tip: hold down shift and right click in the directory to get an "open powershell window here" menu option)?

The weapons etc directories must be accessible from the folder the tool is in.

[geist22]

Yes, the file structure is correct.

I found out what went wrong, the required syntax for the filename is .\filename.txt rather than just filename.txt. Powershell adds the .\ prefix when using tab autocomplete, while the command promt does not, and if you type manually without knowing that you're out of luck.

Anyway, great work! I tested making a custom SOUND.DAT and it works nicely.

[Moburma]

Yeah, that's a Powershell thing. You don't need that if you're using the command prompt, but you do if using Powershell.

[geist22]

It seems you do need it in the command promt, at least I get the aforementioned No such file or directory error when typing the filename without the .\.

[Moburma]

Ahh, you're right actually, it needs the full path every time, I hadn't noticed before as I was used to doing it the powershell way. I will update the documentation