RDCatch Recording: nothing in cut after apparent successful recording

[alexolivan]

Hi.

I've been playing with rdcatch, trying to do some recordings.
My setup is a VM with just Jackd, so only card 0, wich is jack.
I have setup card 0 as a switch device (just to allow for routing as I learnt from RD 3.X)

I have setup a recording event in RDCatch.
I've selected card 0 / port 0
Format is mpeg layer 2 / bitrate 128kb
Switcher Host to NONE

This doesn't work, and following line appears in syslog:

Mar  9 11:45:00 debian caed: invalid twolame parameters, card=0, stream=0, chans=2, samprate=44100  bitrate=0

Bitrate=0 caught my attention... is that right?

I've managed to make it to do something by seting the switcher host to the host itself, then choosing the jackd card switcher...but it seems to trigger a sound loop/feed-back at the input with lots of noise.

No recording to file anywaus whatsoever

[ElvishArtisan]

Confirmed here.

[alexolivan]

I've noticed that tonight's package builds would include a pair of commits related to RDCatch problems, one of them including this...so I've checked this by re-scheduling a recording.
The result was successful.
I've tested with and without autotrim/normalize options, just in case, and no problems (only mpeg2 codec tested)

I'm not sure whether should I close this issue, since may be other commits related to all this queued to be pushed and tested... (I will open another, separated issue, related to yet another one-shot issue regarding recordings in rdcatch, as I thing those are totally unrelated)

[ElvishArtisan]

This specific issue should be fixed by c22a040. That said, in the course of fixing this, I uncovered a bunch of problems lurking in the RDCatch subsystem. Stay tuned for more fixes!

[alexolivan]

OK... so, I think I better close this particular one, so the issues list gets cleaner.

Thanks!

[alexolivan]

Hi again... I'm reopening this.
I left the recording event that I used to perform the testing (successfully?) last time, there on RDCatch, on the test host, so in theory, every day, at the same our, a 5 minutes recording shoud be done.

However, today, I've realized in RDLibrary that the recording cart was 'red' ... so I looked at the cut, and no audio present was reported... and it should.
So, I have tried for three times to re-schedule a 2 minute test record in the future, wait, and see what happens...

From GUI perspective, apparently, it works: RDCatch vumeter at the record deck goes live, and everything seems to be working fine.
From syslog perspective, it seems also to work too...here's the relevant lines of last run:

May 24 16:50:00 debian rdcatchd: record started: deck: 1, event: 215 card: 0, stream: 0, cut: rdcatchd-record-215 length: 119997
May 24 16:50:00 debian caed: LoadRecord - Card: 0  Stream: 0  Coding: 2  Chans: 2  SampRate: 44100  BitRate: 128000  Name: /var/snd/rdcatchd-record-215.wav
May 24 16:50:00 debian caed: Record - Card: 0  Stream: 0  Length: 119997  Thres: 0
May 24 16:50:00 debian rdcatchd: Loaded - Card: 0  Stream: 0
...
May 24 16:52:00 debian caed: UnloadRecord - Card: 0  Stream: 0, Length: 5311488
May 24 16:52:00 debian rdcatchd: record complete: cut 061001_001
May 24 16:52:00 debian ripcd: sent notification: "NOTIFY CART MODIFY 61001" to 239.192.255.72:20539
May 24 16:52:00 debian rdcatchd: created connection 1
May 24 16:52:00 debian ripcd: added new connection 11154
May 24 16:52:00 debian rdcatchd: started import of /var/snd/rdcatchd-record-215.wav to cut 061001_001, id=215
May 24 16:52:04 debian rdcatchd: completed import of /var/snd/rdcatchd-record-215.wav to cut 061001_001, id=215
May 24 16:52:04 debian rdcatchd: deleted file "/var/snd/rdcatchd-record-215.wav"
May 24 16:52:04 debian ripcd: sent notification: "NOTIFY CART MODIFY 61001" to 239.192.255.72:20539

Looks good! ... it even seems it is handling the temp file as it ends generating the cut on the cart...however, nothing ends on the cut.
I've tried removing the autotrim just in case, and alse kept an eye on RDCatch VUMeters to ensure theres some content that should go to the cut.

How could I troubleshoot this?

EDIT 1: Listing /var/snd for record files shows some old recordings there... maybe old troublesome pieces of recordings... I think they're safe to delete:

root@debian:~# ls -l /var/snd/*record*
-rw-rw-r-- 1 rivendell rivendell 1934495 May  2 12:12 /var/snd/rdcatchd-record-218.wav
-rw-rw-r-- 1 rivendell rivendell 3982239 May  2 12:19 /var/snd/rdcatchd-record-219.wav

EDIT 2: Checked wether the temp file does actually grows in size as a new attemtp goes on

root@debian:~# ls -l /var/snd/*record*215.wav
-rw-rw-r-- 1 rivendell rivendell 104126 May 24 17:15 /var/snd/rdcatchd-record-215.wav
root@debian:~# ls -l /var/snd/*record*215.wav
-rw-rw-r-- 1 rivendell rivendell 998174 May 24 17:16 /var/snd/rdcatchd-record-215.wav
root@debian:~# ls -l /var/snd/*record*215.wav
-rw-rw-r-- 1 rivendell rivendell 1942073 May 24 17:17 /var/snd/rdcatchd-record-215.wav

So, apparently, audio (or something) went to the file... but then seems as if the actual import into the cart/cut never happened or failed silently

[ElvishArtisan]

Unable to reproduce the problem here. Any chance that you could post a screenshot of the 'Edit Recording' dialog in rdcatch(1) for the event? That way, I can see the precise parameters you're using. Also: what audio type are you using (HPI, JACK, etc)?

[alexolivan]

Hi Fred... sure.
Ok... though that RDAdmin showed the info you needed...but it does not... RDcatch uses card 0, whic is Jack.
I'm using small VMs for testing, so no hardware involved.
The image I'm using has a switcher configured using regular sound card (so, jack again)

What's most surprising is that actual data ends into the temp file... it very much seems as it fails at the very last step.

EDIT: I removed also the 'normalize' option just to ease the ingest of the recorded audio (which is actually recording)

And it worked!!!!

I have to make further tests to try to have a consistent behaviour... at this point, seems that if normalize AND autotrim are unchecked, the recording works... but I would like to test several times every combination

[alexolivan]

Hi again... I've done some further testing

Here's an screenshot of RDCatch event scrolled right for details of event being shown (Nice feature I didn't notice!!! ... I however notice the trim field is showing -4000dB instead of -40.00dB, and also I see there's no column for the normalization field) regarding codec, channels, bitrate, etc

Early results seem to point at a problem with NORMALIZATION.
If normalization is disabled, recordings do get imported to cart/cut successfully, either with or without the autotrim checkbox.
I have not tried changing normalization level (just in case) and I have it at -8dB... will test wether it does matter

EDIT:
I changed the normalize value to -12dB and repeated the tests with same result: audio ends up in the cut only if the normalization is not checked.

[ElvishArtisan]

Still unable to goose it here.

Could you configure your Syslog setup to record DEBUG level messages and then send me the output during the entire course of the event? That should flag exactly where this is going off the rails.

[alexolivan]

HI Fred... I'm trying to get DEBUG messages from rivendell, but I'm failing to do so.

EDIT:

OK ... I fugured out that /etc/rd.conf defaults to normal user log, and has to be set accordingly to match LOCAL7 facility as provided in rivendell's rsyslog.d conf file!
So, now I have SyslogFacility=23 in rd.conf

The, I took a look at /etc/rsyslog.d/10-rivendell.conf , uncommented the line as suggested:
LOCAL7.DEBUG /var/log/rivendell/debug
Then restarted rivendell and rsyslog, and now I got operation and debug files getting content under /var/log/rivendell

So... Time to repeat the bugged recording -> export test

Related lines on /var/log/rivendell/debug:

May 27 11:01:03 debian ripcd: sent notification: "NOTIFY CATCH_EVENT MODIFY 215" to 239.192.255.72:20539
May 27 11:01:03 debian rdcatchd: removed event 215, Type: recording, Cut: 061001_001
May 27 11:01:03 debian rdcatchd: loading event 215, Type: recording, Cut: 061001_001
May 27 11:01:03 debian rdcatchd: removed event 215, Type: recording, Cut: 061001_001
May 27 11:01:03 debian rdcatchd: loading event 215, Type: recording, Cut: 061001_001
May 27 11:01:21 debian ripcd: added new connection 40
May 27 11:01:21 debian rdxport.cgi: processing RDXPORT_COMMAND_AUDIOSTORE
May 27 11:02:00 debian rdcatchd: record started: deck: 1, event: 215 card: 0, stream: 0, cut: rdcatchd-record-215 length: 119997
May 27 11:02:00 debian caed: LoadRecord - Card: 0  Stream: 0  Coding: 2  Chans: 2  SampRate: 44100  BitRate: 128000  Name: /var/snd/rdcatchd-record-215.wav
May 27 11:02:00 debian caed: Record - Card: 0  Stream: 0  Length: 119997  Thres: 0
May 27 11:02:00 debian rdcatchd: Loaded - Card: 0  Stream: 0

...

May 27 11:04:00 debian caed: UnloadRecord - Card: 0  Stream: 0, Length: 5312512
May 27 11:04:00 debian rdcatchd: record complete: cut 061001_001
May 27 11:04:00 debian ripcd: sent notification: "NOTIFY CART MODIFY 61001" to 239.192.255.72:20539
May 27 11:04:00 debian rdcatchd: created connection 1
May 27 11:04:00 debian ripcd: added new connection 43
May 27 11:04:00 debian rdcatchd: started import of /var/snd/rdcatchd-record-215.wav to cut 061001_001, id=215
May 27 11:04:04 debian rdcatchd: completed import of /var/snd/rdcatchd-record-215.wav to cut 061001_001, id=215
May 27 11:04:04 debian rdcatchd: deleted file "/var/snd/rdcatchd-record-215.wav"
May 27 11:04:04 debian ripcd: sent notification: "NOTIFY CART MODIFY 61001" to 239.192.255.72:20539
May 27 11:04:09 debian rdcatchd: closed connection 1

Related lines on /var/log/rivendell/operations

May 27 11:02:00 debian rdcatchd: record started: deck: 1, event: 215 card: 0, stream: 0, cut: rdcatchd-record-215 length: 119997
May 27 11:02:00 debian caed: LoadRecord - Card: 0  Stream: 0  Coding: 2  Chans: 2  SampRate: 44100  BitRate: 128000  Name: /var/snd/rdcatchd-record-215.wav
May 27 11:02:00 debian caed: Record - Card: 0  Stream: 0  Length: 119997  Thres: 0
May 27 11:04:00 debian caed: UnloadRecord - Card: 0  Stream: 0, Length: 5312512
May 27 11:04:00 debian rdcatchd: record complete: cut 061001_001
May 27 11:04:00 debian rdcatchd: started import of /var/snd/rdcatchd-record-215.wav to cut 061001_001, id=215
May 27 11:04:04 debian rdcatchd: completed import of /var/snd/rdcatchd-record-215.wav to cut 061001_001, id=215

Apparently, it all goes well... however, I got a 'red cart' (no audio in cut) whnever normalize is checked.
I think the kind of debug you refer is more of the DBG kind (to catch some segfault on the process)
I wonder how could I replicate the actual export under debugger to see why normalization is not working for RDCatch recording, but does work if I import audio from an audio mp3 o flac file into a cut (I have done that many times) from RDLibrary, which I suspect does use the same underlying import/export tool.

[ElvishArtisan]

Should be fixed in current 'qt5' branch. Please test!

[alexolivan]

Hi fred.
Tested again, and the bug is somewhere hidden there: I've scheduled right now a short recording, with the only difference of having 'normaliation' enabled:

During this August holidays I've been pushing my mockup VM quite a bit... I've had a ton of fun setting it up kinda close to a real radio (I've even named it :-P), and I've been theese weeks testing a whole setup of automated recordings of certain parts of the grid, together with their automatic upload and publishing on a podcast feed... so far, it works perfect! recordings are done and upload and it is working for weeks, super stable (as far as I've managed to test)... but still, the normalize checkbox must be kept untiked, otherwise, the temp wav audio record never gets to the target cut.

[ElvishArtisan]

'Kay, thanks for testing! I'll pound on it a bit more here.

[ElvishArtisan]

'Kay, I've been testing this heavily on Linux Mint 21 (basically, Ubuntu Jammy with XFCE), using a JACK-based setup. It's been rock solid. I'm not sure where to go from here. Any chance that I could get access to your instance and see this in realtime?

[alexolivan]

Hi! ... The VM is quite isolated, very hard to get SSH
I'm using an VNC through WEB to access it on the hypervisor ... however, I'm thinking on installing some kind of stuff there such as Anydesk or something similar to have a 'conventional' remote access. Will confirm it to you.

Anyhow, chances are that what I'm experiencing being a problem very specific to this install for whatever reason.
The only thing I recall in the past being somehow troublesome was because my settings are not default PCM, but MPEG2:

For the sake of following through involved setup, RDCatch record event looks like this:

And the host RDLibrary settings being:

I don't recall of anything else that may affect the final audio ingest into the Library... If so I'm missing it now.

I've actually checked, during recording, at filesystem level, that actual .wav data is actually recorded in the temp file.. I mean, the actual audio is there. Moreover, without noormalization, it works fine.
So it is something silently failing on the final step, when normalization is activated.
If an exact setup like mine works in another similar system, then my bet it is something related to the packaged audio libraries on Debian Bullseye causing the problem, not Rivendell itself.

Cheers!

EDIT:

• Looking all 3 screenshots at a time (not possible in real RDAdmin) I realize that there may be some conficting settings around., for instance, normalization level.
• I'm thinking about which of the settings has priority. My bet RDLibrary settings do override conflicting settings at RDCatch event.

[ElvishArtisan]

I'm thinking about which of the settings has priority. My bet RDLibrary settings do override conflicting settings at RDCatch event.

The other way around, actually; the settings in 'RDLibrary' are the defaults, which can then be modified on a per-event basis.

The screenshots are very helpful. Any chance you could post the one for RDAdmin->ManageHosts->[hostname]->RDCatch as well?

[ElvishArtisan]

The screenshots are very helpful. Any chance you could post the one for RDAdmin->ManageHosts->[hostname]->RDCatch as well?

No worries; I see you have already! (Time for coffee...)

[alexolivan]

I'm closing this one... I feel we may get into this in the future as I strongly believe (after the Magick feed image issues) that all this could be simply a matter of media libraries on Debian Bullseye being more recent that the ones in CentOS (specially for those using deb-multimedia packages).
I wouldn't be surprise it may appear on Ubuntu 22.04 or future releases.

It is just a matter of being on alert and recall on this, provide eventual future problems or issue reports on blank recordings.
In the mean time, there's no point on having this opened

Thank you for your presence.