Skip to content

refactor: remove fatal error from the constructor of texture. #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

refactor: remove fatal error from the constructor of texture. #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3016ae6
refactor: remove the fetal error from the constructor of the Texture
ChiaoteNi Jun 18, 2024
ba5a7ac
refactor: bypass the input texture as the output one for the exceptio…
ChiaoteNi Jun 19, 2024
870abf3
fix: add missing signal and remove the unrequired one
ChiaoteNi Jun 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 31 additions & 9 deletions framework/Source/BasicOperation.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,6 @@ open class BasicOperation: ImageProcessingOperation {

public func newTextureAvailable(_ texture: Texture, fromSourceIndex: UInt) {
let _ = textureInputSemaphore.wait(timeout:DispatchTime.distantFuture)
defer {
textureInputSemaphore.signal()
}

inputTextures[fromSourceIndex] = texture

if (UInt(inputTextures.count) >= maximumInputs) || activatePassthroughOnNextFrame {
Expand All @@ -76,10 +72,23 @@ open class BasicOperation: ImageProcessingOperation {
uniformSettings["aspectRatio"] = firstInputTexture.aspectRatio(for: outputRotation)
}

guard let commandBuffer = sharedMetalRenderingDevice.commandQueue.makeCommandBuffer() else {return}
guard
let commandBuffer = sharedMetalRenderingDevice.commandQueue.makeCommandBuffer(),
let outputTexture = Texture(
device:sharedMetalRenderingDevice.device,
orientation: .portrait,
width: outputWidth,
height: outputHeight,
timingStyle: firstInputTexture.timingStyle
)
else {
assertionFailure("CommandBuffer or Texture creation failed")
removeTransientInputs()
textureInputSemaphore.signal()
updateTargetsWithTexture(firstInputTexture)
return
}

let outputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation: .portrait, width: outputWidth, height: outputHeight, timingStyle: firstInputTexture.timingStyle)

guard (!activatePassthroughOnNextFrame) else { // Use this to allow a bootstrap of cyclical processing, like with a low pass filter
activatePassthroughOnNextFrame = false
// TODO: Render rotated passthrough image here
Expand All @@ -95,8 +104,21 @@ open class BasicOperation: ImageProcessingOperation {
if let alternateRenderingFunction = metalPerformanceShaderPathway, useMetalPerformanceShaders {
var rotatedInputTextures: [UInt:Texture]
if (firstInputTexture.orientation.rotationNeeded(for:.portrait) != .noRotation) {
let rotationOutputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation: .portrait, width: outputWidth, height: outputHeight)
guard let rotationCommandBuffer = sharedMetalRenderingDevice.commandQueue.makeCommandBuffer() else {return}
guard
let rotationCommandBuffer = sharedMetalRenderingDevice.commandQueue.makeCommandBuffer(),
let rotationOutputTexture = Texture(
device:sharedMetalRenderingDevice.device,
orientation: .portrait,
width: outputWidth,
height: outputHeight
)
else {
assertionFailure("CommandBuffer or Texture creation failed")
removeTransientInputs()
textureInputSemaphore.signal()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to call this?

https://github.com/cardinalblue/GPUImage3/pull/22/files#diff-ff11237abc08f188e5a08bf94b2e6d829bf94f01fe50e2daf5a011c809415a72R55-R57

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we should, and I notice it shouldn't be invoke inside the defer
Let we add and remove them in the next commit, thanks!

Copy link
Member

@yyjim yyjim Jun 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure if this correct or not. The L138 call also wait.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling it in defer seems reasonable to me since it calls wait at the beginning.

it only wants to run one at the time.

Copy link
Author

@ChiaoteNi ChiaoteNi Jun 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's weird...sorry I didn't notice this 🤔
I'll go back to check this later after I update the implementation for the PR of the face mask.

updateTargetsWithTexture(firstInputTexture)
return
}
rotationCommandBuffer.renderQuad(pipelineState: sharedMetalRenderingDevice.passthroughRenderState, uniformSettings: uniformSettings, inputTextures: inputTextures, useNormalizedTextureCoordinates: useNormalizedTextureCoordinates, outputTexture: rotationOutputTexture)
rotationCommandBuffer.commit()
rotatedInputTextures = inputTextures
Expand Down
4 changes: 2 additions & 2 deletions framework/Source/Camera.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,8 +217,8 @@ public class Camera: NSObject, ImageSource, AVCaptureVideoDataOutputSampleBuffer
outputWidth = bufferWidth
outputHeight = bufferHeight
}
let outputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation:.portrait, width:outputWidth, height:outputHeight, timingStyle: .videoFrame(timestamp: Timestamp(currentTime)))
let outputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation:.portrait, width:outputWidth, height:outputHeight, timingStyle: .videoFrame(timestamp: Timestamp(currentTime)))!

convertYUVToRGB(pipelineState:self.yuvConversionRenderPipelineState!, lookupTable:self.yuvLookupTable,
luminanceTexture:Texture(orientation: self.orientation ?? self.location.imageOrientation(), texture:luminanceTexture),
chrominanceTexture:Texture(orientation: self.orientation ?? self.location.imageOrientation(), texture:chrominanceTexture),
Expand Down
2 changes: 1 addition & 1 deletion framework/Source/ImageGenerator.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ public class ImageGenerator: ImageSource {

public init(size:Size) {
self.size = size
internalTexture = Texture(device:sharedMetalRenderingDevice.device, orientation:.portrait, width:Int(size.width), height:Int(size.height), timingStyle:.stillImage)
internalTexture = Texture(device:sharedMetalRenderingDevice.device, orientation:.portrait, width:Int(size.width), height:Int(size.height), timingStyle:.stillImage)!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Is it safe to use force unwrap here?

}

public func transmitPreviousImage(to target:ImageConsumer, atIndex:UInt) {
Expand Down
2 changes: 1 addition & 1 deletion framework/Source/MovieInput.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ public class MovieInput: ImageSource {

if let concreteLuminanceTextureRef = luminanceTextureRef, let concreteChrominanceTextureRef = chrominanceTextureRef,
let luminanceTexture = CVMetalTextureGetTexture(concreteLuminanceTextureRef), let chrominanceTexture = CVMetalTextureGetTexture(concreteChrominanceTextureRef) {
let outputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation:.portrait, width:bufferWidth, height:bufferHeight, timingStyle:.videoFrame(timestamp:Timestamp(withSampleTime)))
let outputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation:.portrait, width:bufferWidth, height:bufferHeight, timingStyle:.videoFrame(timestamp:Timestamp(withSampleTime)))!

convertYUVToRGB(pipelineState:self.yuvConversionRenderPipelineState, lookupTable:self.yuvLookupTable,
luminanceTexture:Texture(orientation:.portrait, texture:luminanceTexture),
Expand Down
2 changes: 1 addition & 1 deletion framework/Source/MovieOutput.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ public class MovieOutput: ImageConsumer, AudioEncodingTarget {
if (Int(round(self.size.width)) != texture.texture.width) && (Int(round(self.size.height)) != texture.texture.height) {
let commandBuffer = sharedMetalRenderingDevice.commandQueue.makeCommandBuffer()

outputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation: .portrait, width: Int(round(self.size.width)), height: Int(round(self.size.height)), timingStyle: texture.timingStyle)
outputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation: .portrait, width: Int(round(self.size.width)), height: Int(round(self.size.height)), timingStyle: texture.timingStyle)!

commandBuffer?.renderQuad(pipelineState: renderPipelineState, inputTextures: [0:texture], outputTexture: outputTexture)
commandBuffer?.commit()
Expand Down
9 changes: 7 additions & 2 deletions framework/Source/Operations/CBRescaleEffect.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,19 @@ public class CBRescaleEffect: BasicOperation {
}

let outputSize = calculateTargetSize(from: texture)
let outputTexture = Texture(
guard let outputTexture = Texture(
device: sharedMetalRenderingDevice.device,
orientation: texture.orientation,
pixelFormat: texture.texture.pixelFormat,
width: Int(outputSize.width),
height: Int(outputSize.height),
timingStyle: texture.timingStyle
)
) else {
assertionFailure("CommandBuffer or Texture creation failed")
removeTransientInputs()
updateTargetsWithTexture(texture)
return
}

inputTextures[0] = texture
internalRenderFunction(commandBuffer: commandBuffer, outputTexture: outputTexture)
Expand Down
10 changes: 8 additions & 2 deletions framework/Source/Operations/HistogramEqualization.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,19 @@ public class HistogramEqualization: ImageProcessingOperation {
width: CGFloat(frameTexture.width),
height: CGFloat(frameTexture.height)
)
let outputTexture = Texture(
guard let outputTexture = Texture(
device: sharedMetalRenderingDevice.device,
orientation: texture.orientation,
pixelFormat: texture.texture.pixelFormat,
width: Int(size.width),
height: Int(size.height),
timingStyle: texture.timingStyle
)
) else {
assertionFailure("CommandBuffer or Texture creation failed")
removeTransientInputs()
updateTargetsWithTexture(texture)
return
}

inputTexture = texture
renderer.render(
Expand All @@ -51,6 +56,7 @@ public class HistogramEqualization: ImageProcessingOperation {
)
inputTexture = nil

removeTransientInputs()
updateTargetsWithTexture(outputTexture)
}
}
16 changes: 8 additions & 8 deletions framework/Source/PictureOutput.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ public class PictureOutput: ImageConsumer {
storedTexture = texture
}

if let imageCallback = imageAvailableCallback {
let cgImageFromBytes = texture.cgImage()
if let imageCallback = imageAvailableCallback,
let cgImageFromBytes = texture.cgImage() {

// TODO: Let people specify orientations
#if canImport(UIKit)
let image = UIImage(cgImage:cgImageFromBytes, scale:1.0, orientation:.up)
Expand All @@ -61,15 +61,15 @@ public class PictureOutput: ImageConsumer {
#endif

imageCallback(image)

if onlyCaptureNextFrame {
imageAvailableCallback = nil
}
}
if let imageCallback = encodedImageAvailableCallback {
let cgImageFromBytes = texture.cgImage()

if let imageCallback = encodedImageAvailableCallback,
let cgImageFromBytes = texture.cgImage() {

let imageData:Data
#if canImport(UIKit)
let image = UIImage(cgImage:cgImageFromBytes, scale:1.0, orientation:.up)
Expand Down
26 changes: 21 additions & 5 deletions framework/Source/Texture.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,16 @@ public class Texture {
self.timingStyle = timingStyle
}

public init(device:MTLDevice, orientation: ImageOrientation, pixelFormat: MTLPixelFormat = .bgra8Unorm, width: Int, height: Int, mipmapped:Bool = false, timingStyle: TextureTimingStyle = .stillImage) {
public init?(device:MTLDevice, orientation: ImageOrientation, pixelFormat: MTLPixelFormat = .bgra8Unorm, width: Int, height: Int, mipmapped:Bool = false, timingStyle: TextureTimingStyle = .stillImage) {
let textureDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .bgra8Unorm,
width: width,
height: height,
mipmapped: false)
textureDescriptor.usage = [.renderTarget, .shaderRead, .shaderWrite]

guard let newTexture = sharedMetalRenderingDevice.device.makeTexture(descriptor: textureDescriptor) else {
fatalError("Could not create texture of size: (\(width), \(height))")
assertionFailure("Could not create texture of size: (\(width), \(height))")
return nil
}

self.orientation = orientation
Expand Down Expand Up @@ -113,10 +114,25 @@ extension Texture {
}

extension Texture {
func cgImage() -> CGImage {
func cgImage() -> CGImage? {
// Flip and swizzle image
guard let commandBuffer = sharedMetalRenderingDevice.commandQueue.makeCommandBuffer() else { fatalError("Could not create command buffer on image rendering.")}
let outputTexture = Texture(device:sharedMetalRenderingDevice.device, orientation:self.orientation, width:self.texture.width, height:self.texture.height)
guard
let commandBuffer = sharedMetalRenderingDevice.commandQueue.makeCommandBuffer()
else {
assertionFailure("Could not create command buffer on image rendering.")
return nil
}
guard
let outputTexture = Texture(
device:sharedMetalRenderingDevice.device,
orientation: orientation,
width: texture.width,
height: texture.height
)
else {
assertionFailure("MTLDevice makeTexture failed")
return nil
}
commandBuffer.renderQuad(pipelineState:sharedMetalRenderingDevice.colorSwizzleRenderState, uniformSettings:nil, inputTextures:[0:self], useNormalizedTextureCoordinates:true, outputTexture:outputTexture)
commandBuffer.commit()
commandBuffer.waitUntilCompleted()
Expand Down