Pro Tools crashes or outputs garbage if input and output sample types don't match

[dechamps]

This is a spin-off from #84.

A user saw Pro Tools crash at startup when using sampleType = "Int24". The crash occurs on the first stream callback, which seems consistent with an error in buffer manipulation.

One thing that's interesting in this particular configuration is that the sample type is only set to int24 on the output. The input is using the default float sample type. This is interesting because asymmetric input/output sample types are highly unusual and not well-tested - not just in FlexASIO, but in host applications as well.

[dechamps]

@Kurausukun if you have time, do you mind answering the following questions:

1. Can you confirm it's the sampleType = "Int24" option that's causing Pro Tools to crash? That's not entirely clear from your last Pro Tools WASAPI AUDCLNT_E_DEVICE_IN_USE error #84 (comment)
2. Does Pro Tools still crash if you set the sampleType = "Int24" option in both the [output] and [input] sections, as opposed to just [output]?

[dechamps]

Reproducing locally with Pro Tools First 2019.6.0, the following configuration:

[output]
sampleType = "Int24"

Doesn't make Pro Tools crash, but the sound signal on the output is garbage. Same if only the input is set to Int24.

Using Int16 instead of Int24 makes Pro Tools report an access violation.

Setting both input and output to Int24 works fine.

Clearly this is a bug where either FlexASIO or Pro Tools is failing to deal correctly with channels using different sample types.

[dechamps]

I'm unable to reproduce with FlexASIOTest, REW, and Max MSP. I am therefore going to treat this as a Pro Tools bug.

[dechamps]

I just realized that this problem can also be triggered by automatic sample type detection in WASAPI Exclusive mode if input and output devices have different "native" sample types, which is way more problematic because that's a way more typical scenario that users are more likely to hit.

[dechamps]

Here's the simplest possible setup that can still reproduce:

bufferSizeSamples = 1024

[input]
channels = 1
sampleType = "Int16"

[output]
channels = 1

[dechamps]

While investigating this I also found #91 - it turns out that there are problems with Pro Tools and Int24 sample types, even if only one direction is used. The present issue (sample type mismatch) can still be reproduced with Int16 though, so these are likely still two separate issues.

[dechamps]

I can also reproduce in latest Pro Tools 2020.9 (which just came out).

[Kurausukun]

Sorry for abandoning this, I honestly kind of forgot about it after the original issue was resolved. I'm still willing to help if necessary. It looks like you tested everything locally, but let me know if you still need something.

[dechamps]

@Kurausukun At this point, my working hypothesis is that this problem (as well as #89, #90, and #91) are caused by bugs in Pro Tools, not FlexASIO. I tried to talk to Avid but couldn't get them to listen. If you have a paid version of Pro Tools then you might be able to open a support case with Avid and get them to follow through. Otherwise, I'm afraid there's nothing else I can do here.

Also, it's worth noting that I did all the reproducing in Pro Tools First. Maybe other variants of Pro Tools behave differently.

[Kurausukun]

I also only have Pro Tools First; I only use it for a specific plugin that is exclusive to it, so I don't have any reason to spend money on a premium version.

I also agree with you that it's likely a problem with Pro Tools itself. However, I have found settings that work for me, and the other problems were caused by outside factors, so I am happy with the current results--I have achieved what my original goal was. I'm happy to just close the issues since there's an easy workaround and there's good evidence it isn't a problem with FlexASIO itself.