Implement Scanning of Objects

[aferriss]

Implement the ARObjectScanningConfiguration for an arkit session. This is it's own configuration type and should only be used for apps that are actively doing scanning (says apple).

In order to scan something you need to first define a boundary cube around it. We need a transform representing a local coordinate system of the object. An extent measuring the width, height, and depth of the object. Lastly, a center point relative to the origin of the transform. In the examples I've seen they use a gui with a semi transparent cube alongside the horizontal plane detector. This way you can position your cube directly on what you'd like to scan. Once you have all that you can call this very long method. That method should return the reference object that can be saved and used to detect the object within another app.

I think we should definitely include a sample app that just does scanning as a utility. The app should probably let users save both the reference object file and the usdz model generated during the scan (Though I haven't yet seen how to access the raw 3d information from the reference object).

Here's apple's sample app and another demo from unity

[mayakraft]

Hey all,

Spending the last couple days with this, and I have the skeleton for this example scanning app you're describing Adam. It can switch between the two configurations: ARObjectScanningConfiguration and ARWorldTrackingConfiguration, it sets a wireframe cube around the scan area, and will successfully scan a 3D model and save it into memory (I think, can't test it yet). Your code got me 99% of the way there. thank you.

My code crashes when I try to use the object. I need to get the ARReferenceObjects into an NSSet, as that is what ARWorldTrackingConfiguration's detectionObjects is expecting. Maybe I'm missing something fundamental about how Objective C works..

inside the function createScan
inside the completion handler of session createReferenceObjectWithTransform:
store the reference object in a set, for now, a public class member variable:

[session createReferenceObjectWithTransform:transform center:simCenter extent:simExtent completionHandler:^(ARReferenceObject * referenceObj, NSError * err){
	if(!err){
		scanSet = [NSSet setWithObject:referenceObj];
	}
}];

then in my ofApp.mm, simply reading from this set will crash it

worldConfiguration.detectionObjects = processor->detector->scanSet;

with an error Thread 1: EXC_BAD_ACCESS (code=1, address=0x207fb8110)

Does this mean the NSCoder is having trouble re-assembling an object from the memory inside this set?

At the moment I'm not using the ScannedObject struct type, It seems reasonable to bypass it for the moment. Oh, and I also tried wrapping a dispatch_async(dispatch_get_main_queue(), ^(){ inside the completion handler for Apple's createReferenceObjectWithTransform callback.

the "Bad Access" is almost definitely not pointing invalid memory, but maybe something inside the object is invalid..?

Thanks friends! Hopefully can wrap up this example scan app and share it.

[aferriss]

Hey there! I only ever got as far as saving the object. Once I started to make the example app I got distracted with some other projects. Assuming you saw my branch, but I never actually tried to reload and scan anything. It's all untested so it's very likely that I did something wrong in the implementation (I'm also just learning obj-c).

I do also recall being confused about how to be sure that there would be something in the collection of scans after the completion handler returned. I assumed checking to make sure that the set wasn't empty before trying to access it would be good enough but maybe not....

Wish I had better information for you!

[mayakraft]

hey thanks! yeah no worries I feel like I'm almost there..

yep, using your branch.

Yeah I feel like the code you wrote is good! After a scan, Apple's function successfully calls the completion handler, and there's no error, I expect the ARReferenceObject to be fully formed.

Yeah you have the scanned objects being collected into a C++ vector which should be no problem! Maybe a NSSet would be nice, somewhere, maybe it should all be hidden from the coder.. On the other end of it all setting up a new configuration after scanning goes something like this:

ARWorldTrackingConfiguration *worldConfiguration = [ARWorldTrackingConfiguration new];
worldConfiguration.worldAlignment = ARWorldAlignmentGravity;
// this will add any 2D reference images to the detection
worldConfiguration.detectionImages = [NSSet setWithObject:refImage];
// THIS is where we add the object that was just scanned
worldConfiguration.detectionObjects = [NSSet setWithObject:refObject];

(and in my code, the last line is where the crash is happening)

[aferriss]

One thing that helped me a bit for getting some of this going was looking at apple's example scanning app. However it is in swift so it takes some work to convert into obj-c. The relevant code doesn't shine much light on your problem though. They do this

 func setReferenceObject(_ object: ARReferenceObject, screenshot: UIImage) {
        referenceObject = object
        previewImage = screenshot
        detections = 0
        lastDetectionDelayInSeconds = 0
        averageDetectionDelayInSeconds = 0
        
        self.detectedObject = DetectedObject(referenceObject: object)
        self.sceneView.scene.rootNode.addChildNode(self.detectedObject!)
        
        self.lastDetectionStartTime = Date()
        
        let configuration = ARWorldTrackingConfiguration()
        configuration.detectionObjects = [object]
        self.sceneView.session.run(configuration)
    }