Lossy flac-compressed files: issue with sox

[davidvankemenade]

Checklist

• ✅ I have searched the issues page for any duplicate issues open or closed and confirmed that this bug has not been reported before.
• ✅ I have tested the issue with the current build.
• ✅ I have attached log files, uploaded sample data, and commands used so that the issue can be easily reproduced by the developers.

Bug Description

Decode.exe doesn't output anything when the input file is a long, flac-compressed file.
(It works fine when the input file is a long, uncompressed 16-bit file.)
(It works fine when the input file is a short, flac-compressed file.)

Steps to Reproduce

1. Start DdD capture of a 16-bit raw file for 195 minutes
2. Once the capture has been running for a few seconds, start the tape (VHS or Betamax)
3. Once the capture has been completed, compress the file using sox.exe --rate 40000 --bits 16 --channels 1 --encoding signed --type raw capture.s16" --bits 8 --rate 20000 --channels 1 --type flac --compression 8 compressed.flac sinc -n 2500 0-9650
4. Decode using decode.exe vhs --sub_deemphasis --frequency 20 --level_detect_divisor 1 --fallback_vsync --recheck_phase --pal --threads 4 --tape_format Betamax compressed.flac decodedfile

Expected Behaviour

Decode.exe starts decoding as normal. Four files will be groing as decode process runs: .log, .json, .tbc, _chroma.tbc.

Actual Behaviour

After about 20 seconds, decode.exe fails with

fd:: Invalid data found when processing input
Traceback (most recent call last):
  File "vhsdecode\process.py", line 285, in build_json
TypeError: 'NoneType' object is not subscriptable
Cannot build json: 'NoneType' object is not subscriptable

Completed without handling any frames.
Traceback (most recent call last):
  File "vhsdecode\process.py", line 285, in build_json
TypeError: 'NoneType' object is not subscriptable
Cannot build json: 'NoneType' object is not subscriptable

Output files .log, .tbc, _chroma.tbc are all 0 bytes. No .json output file is available.

Environment

• Decode version: 0.2.5
• Operating System: Windows 10
• Hardware Used: Domesday Duplicator

Additional Information

• Behaviour is the same for both Betamax and VHS
• When following the exact same process but with a capture length of 1 minute, everything works perfectly fine.
• In the 1-minute scenario, it doesn't matter whether the tape is already running when capture starts, or the capture starts a few seconds before the tape starts running.
• The long file fails to decode in Ubuntu as well, using commit d43225a. In this case, the error output starts with pipe:: Invalid data found when processing input
• Assumption: As very long files work fine if they are not sox-compressed, my feeling is that something goes wrong during sox compression. Input .s16 file size ~ 900 GByte, .flac file size ~90 GByte.

Is it related to tbc-video-export?

No

[davidvankemenade]

I've looked into this further and it seems the issue is sox-related. I tried to de-compress from .flac to .s16 using sox:
sox.exe --bits 8 --rate 20000 --channels 1 --type flac compressed.flac --rate 40000 --bits 16 --channels 1 --encoding signed --type raw de-compressed.s16

This works with short files and fails with long ones:
sox.exe FAIL formats: bad input format for file compressed.flac: data encoding or sample size was not specified

Is there any way to perform lossy compression without using sox?

[harrypm]

There was issues with SoX on windows so its scripts were removed until it gets figured out, we did not want users care free down sampling files to broken files, currently GNU radio would be your best bet to make proper down-sampled files.

[davidvankemenade]

Thanks for your response @harrypm. I'm not familiar with GNU radio yet. Is it command-line scriptable?

In the meantime, I've found the following workaround: using sox.exe for the lossy compression, then pipe into flac.exe for compression.

sox.exe --no-show-progress --rate 40000 --bits 16 --channels 1 --encoding signed --type raw raw capture.s16 --bits 8 --rate 20000 --channels 1 --type raw - sinc -n 2500 0-9650 | flac.exe - --endian=little --channels=1 --bps=8 --sample-rate=20000 --sign=signed --best -o compressed.flac

This seems to work OK.

For faster processing, it would be even better if FlaLDF supported 8 bits (currently 16 only) and non-standard bitrates (e.g. 20000). I'm not sure if there's a fundamental reason why that wouldn't be possible, or how to request this enhancement.

[davidvankemenade]

Closing issue as workaround is available by using sox.exe in conjunction with flac.exe