backport CoreML features to macos < 14

[glaszig]

i'm on 12.7 and [MLModel predictionFromFeatures:] versions with completion handler are only available on macos 14 which is a little too cutting edge for me.

before this i'd get:

whisper-encoder-impl.m:183:17: error: no visible @interface for 'MLModel'
declares the selector 'predictionFromFeatures:completionHandler:'
    [self.model predictionFromFeatures:input completionHandler:^(id<MLFeatureProvider> prediction, NSError *predictionError) {
     ~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[danbev]

Should the prediction/inference perhaps be run on a background thread and not the main thread as it being done here?

[glaszig]

that was my first impulse as well but the original documentation doesn't state anything about parallelism/threading. then it occurred to me that coreml itself decides where to run (cpu or gpu) 0 and the completionHandler variants are only to enable swift's async/await 1.

that is why i decided against active concurrency management in those methods. async-marked methods in swift actually get their handler-equivalent auto-generated. a difference may be that they use dispatch queues instead of operations though.

[danbev]

I'm just a little concerned that doing the inference on the main thread might possibly lock/stall an application while the inference is running which could be substantial. Would you be able to try something like the following and see if this works:

[NSOperationQueue.new addOperationWithBlock:^{
    id<MLFeatureProvider> prediction = [self predictionFromFeatures:input error:&error];
    
    dispatch_async(dispatch_get_main_queue(), ^{
        completionHandler(prediction, error);
    });
}];

[danbev]

Should the prediction/inference perhaps be run on a background thread and not the main thread as it being done here?