v2.6.0.118490

Video Effects SDK 2.6

Release 2.6.0 for Windows and macOS. Versions 2.4 and 2.5 were iOS-only releases of the Swift package.

IPipeline and IReplacementController are thread-safe

Any method may be called from any thread without synchronization, except that IPipeline::process must not be called while another process() call is running. Loading a model or creating the engine no longer holds up process().
The same holds for the Obj-C/Swift Pipeline: only one of its process methods may run at a time, and ReplacementController.background may be set from any thread.
Enabling an effect may take a while (it may load a model), so avoid calling enable* methods on the UI thread. Parameter setters such as setColorCorrectionPower are cheap.

Effects

• New feature Noise Suppression: Improves video quality by reducing grainy noise in the video and keeps noisy regions stable instead of letting them shimmer from frame to frame. See IPipeline::enableNoiseSuppression.
• New feature Portrait Lighting: Highlights the person in the video, with separate controls of the light strength and of the background darkening. See IPipeline::enablePortraitLighting, setPortraitLightingParam/getPortraitLightingParam with PortraitLightingParam.
• Reworked Background Blur: faster and aligned with the Web SDK.
• Reworked Beautification.
• Reworked Color Correction for more natural results: automatic scene analysis adapts the image to the lighting and color balance, and manual controls of white balance, exposure, temperature, tint, vibrance and contrast fine-tune it. See IPipeline::setColorCorrectionParam/getColorCorrectionParam with ColorCorrectionParam.
• Reworked Low Light: brightens dark scenes without an ML model.

Changes

• New segmentation models, better and faster: the Quality preset is about 1.75x faster (3.0 ms to 1.7 ms per frame on our test CPU).
• DirectX: NV12 conversion is performed on GPU now. process() of an NV12 frame on our test discrete GPU (with background replacement enabled): 1920x1080 3.4 ms instead of 15.7 ms, 3840x2160 5.0 ms instead of 54 ms, 7680x4320 12.5 ms instead of 230 ms.
• Obj-C/Swift: Added the ability to create Frame with CVPixelBuffer, see FrameFactory.newFrame(pixelBuffer:metalCompatible:).
• Windows: Improved performance of IReplacementController::setBackgroundImage: the background is uploaded to GPU without a copy on CPU, NV12 backgrounds are converted on GPU.
• Apple: Improved performance of ReplacementController.background: a CVPixelBuffer background is used by Metal directly, without a copy, when possible; other frames are converted once per change.
• Apple: The Metal pipeline supports RGBA frames created from memory; the processed frame is in memory too, its toCVPixelBuffer returns nil.
• Added Color Range and Color Matrix (BT.601, BT.709) properties in IFrame: IFrame::setColorRange with ColorRange, IFrame::setColorMatrix with ColorMatrix. On Apple they are taken from the CVPixelBuffer.
• Apple: Added MLBackend::mlBackendGPU, enabled by default: Core ML may run segmentation on the GPU too. To keep the CPU and the Neural Engine only, pass mlBackendCPU | mlBackendAppleNeuralEngine to IPipelineConfiguration::setSegmentationMLBackends (Obj-C/Swift: isSegmentationOnGPUEnabled false).
• Frame memory is re-used now: output frames of IPipeline::process() and the copies that IFrameFactory::createBGRA/createNV12 make when makeCopy is true come from a pool (on Apple, a CVPixelBufferPool).
• New error codes in PipelineErrorCode: notSupportedByBackend, enabledUnsupportedFeatures and invalidUsage.
• Windows and macOS: The SDK can be linked directly: sdk_factory.h declares createSDKFactory(), tsvb.lib is shipped. Loading the library at runtime still works.
• Obj-C/Swift: Added enableNoiseSuppression, enablePortraitLighting, effect settings as typed properties colorCorrection (ColorCorrectionSettings) and portraitLighting (PortraitLightingSettings), and isSegmentationOnGPUEnabled.
• Obj-C/Swift: Added read-only isBlurBackgroundEnabled, blurBackgroundPower and isReplaceBackgroundEnabled.
• The API reference is available at https://effectssdk.ai/sdk/desktop/docs/ (C++) and https://effectssdk.ai/sdk/ios/documentation/tsvb/ (Obj-C/Swift).
• Apple: Removed excess dependencies and improved model packing: size of the binaries drastically reduced. macOS libtsvb.dylib: arm64 14.0 MB to 2.05 MB, x86_64 17.2 MB to 2.08 MB.

Breaking changes

• Misspelled enableColorCorrectionWithReferance deprecated. Use enableColorCorrectionWithReference.
• Removed the segmentationPresetSpeed and segmentationPresetLightning values of SegmentationPreset (Obj-C/Swift: speed and lightning of SegmentationPreset).
• Removed the old background denoise effect, replaced by Noise Suppression: enableDenoiseBackground, disableBackgroundDenoise, setDenoisePower/getDenoisePower, setDenoiseWithFace/getDenoiseWithFace and getDenoiseBackgroundState of IPipeline (Obj-C/Swift: enableDenoiseBackground, disableDenoiseBackground, denoisePower and denoiseWithFace).
• Beautification, Color Correction, Low Light do not work on CPU anymore: enabling them on the CPU backen...

v2.3.1.115482

Changes:
Added offline authorization

Release v2.3.0.110702

Changes:

• Web Authorization instead of watermark;
• Added support of Neural Engine for Apple systems;
• New supported CVPixelBuffer formats;
• Faster Background Blur;
• Support of metal compatible CVPixelBuffer;
• Faster NV12 on MacOS.

Release v2.2.1.108022

Bug Fix Release

Fixed:

• Issue of Color Grading when used a high-res image in 32-bit build (Win);
• Balanced Preset for 32-bit build (Win);
• Issue when Denoise and Sharpness enabled same time (macOS);
• Issue when selected a high-res background image for Background Replacement while Denoise enabled (Win);
• Edge artifacts when enabled CPU backed Sharpness (Linux);
• Rare issue when Blur, Sharpness, and Color Correction effects enabled in same time (macOS);
• Performance issue of Beautification and Smart Zoom enabled in same time (Win);

Release v2.2.0.107661

Changes:

• Add Sharpening Effect
• Stability improvements

Release v2.1.0.93319

• Low Light Adjustment
• Other improvements

v2.0.11.86819

Publish the Video Effects SDK desktop sample