Gen2 apriltag

[csaba-luxonis]

WIP. Currently it runs on full frame with ~6 fps, with image manip downscaled it could be 30+ fps but it does not support RAW8 yet (crashes).

[Luxonis-Brandon]

Sweet! Thanks! Nice to at least have the initial version running in Gen2!

[Luxonis-Brandon]

Since I subscribe in the pics or it didn't happen philosophy here is a video showing this initial version running:

[GOBish]

Thanks for everyone’s work on this - any updates on gen2 working at a higher frame rate @csaba-luxonis?

[Luxonis-Brandon]

So as of now this is running at full frame resolution, which makes it slower. @csaba-luxonis is working to integrate this with the Image Manip node such that a lower resolution can be passed into the April Tags node, allowing higher framerates.

[GOBish]

Thanks for all your work on this.
I got the two examples to run but they go super slow for me (< 2 fps). Also, when I put an apriltag in view of the camera (using TAG_STAND41H12), the program crashes immediately and I get the error:

~/depthai-python/examples$ python3 apriltag.py
Traceback (most recent call last):
File "apriltag.py", line 59, in
xmin = int(aprilTag.p.x)
AttributeError: 'depthai.AprilTags' object has no attribute 'p'

update: this happens no matter what type of Apriltag it is, i.e. TAG36H11)

[Luxonis-Brandon]

Interesting. Sorry about the trouble and thanks for the details. So @csaba-luxonis is planning another update today or early tomorrow. I think it would be best to try that, as it has a TON of improvements and optimizations.

[GOBish]

Great, thanks for the update. Also, it seems to need depthai=2.5 and won't work for me with 2.8

[Luxonis-Brandon]

Yes actually it needs the specific depthai referenced by the requirements.txt in ./examples in this branch (at least for now, until it is merged to mainline).

[GOBish]

FYI, changing

xmin = int(aprilTag.p.x)
ymin = int(aprilTag.p.y)
xmax = xmin + int(aprilTag.p.width)
ymax = ymin + int(aprilTag.p.height)

to

xmin = int(aprilTag.points.x)
ymin = int(aprilTag.points.y)
xmax = xmin + int(aprilTag.points.width)
ymax = ymin + int(aprilTag.points.height)

made it work.

[Luxonis-Brandon]

Thanks for the heads up. @csaba-luxonis - are the pushes you just did OK to run with now?

[csaba-luxonis]

Thanks for the heads up. @csaba-luxonis - are the pushes you just did OK to run with now?

Yes it is all ok now, thanks @GOBish for the catch.

[csaba-luxonis]

Right now it migth not be capable to run it in parallel with CNN or stereo due to resource allocation limitation. Working on it together with further speed optimization.

[dhirajdhule]

@csaba-luxonis @Luxonis-Brandon do you have any ETA on when will this be released at a lower resolution, higher FPS? We are happy with processing on 640x360 image on the host side at the moment. If OAK-1 is able to do meet that at 15 FPS, we are good. Eagerly waiting for this.

[dhirajdhule]

Apologies for the digression. Any plans to support ArUco tags in the future?

Sorry for my ignorance; but is it possible for others (outside luxonis) to add this support? If yes, will this be like 'a step by step implementation using basic opencv functions', or will it be something like 'directly exporting opencv-contrib modules that provide ArUco support'?

[Erol444]

Apologies for the digression. Any plans to support ArUco tags in the future?

Sorry for my ignorance; but is it possible for others (outside luxonis) to add this support? If yes, will this be like 'a step by step implementation using basic opencv functions', or will it be something like 'directly exporting opencv-contrib modules that provide ArUco support'?

I'm not sure if aruco tags are on TODO, but it wouldn't be possible to add support for this on the device. FW itself is closed-source (Intel's NDA), and opencv functions don't actually run on the device. And just to give you an idea, implementation is done in c++ and uses SHAVE cores to perform HW accelerated vector computations, so it's far from easy/using basic opencv functions. That's why it takes a few weeks to get such feature off the ground.

[dhirajdhule]

Apologies for the digression. Any plans to support ArUco tags in the future?
Sorry for my ignorance; but is it possible for others (outside luxonis) to add this support? If yes, will this be like 'a step by step implementation using basic opencv functions', or will it be something like 'directly exporting opencv-contrib modules that provide ArUco support'?

I'm not sure if aruco tags are on TODO, but it wouldn't be possible to add support for this on the device. FW itself is closed-source (Intel's NDA), and opencv functions don't actually run on the device. And just to give you an idea, implementation is done in c++ and uses SHAVE cores to perform HW accelerated vector computations, so it's far from easy/using basic opencv functions. That's why it takes a few weeks to get such feature off the ground.

Thanks for the quick and concise response. Very helpful.

[r1b4z01d]

Correct me if I am wrong but this is just doing tag detection and not pose estimation.

[saching13]

Correct me if I am wrong but this is just doing tag detection and not pose estimation.

As of now yes.
The next step would be to use the corners of the detected tag, the size of the marker and calibration information to obtain the pose. This is not implemented on the FW yet.

[r1b4z01d]

@saching13 thank you. What family of aprilTags is this detecting?

[saching13]

@saching13 thank you. What family of aprilTags is this detecting?

Here are the supported tag types

[dhirajdhule]

@csaba-luxonis @Luxonis-Brandon do you have any ETA on when will this be released at a lower resolution, higher FPS? We are happy with processing on 640x360 image on the host side at the moment. If OAK-1 is able to do meet that at 15 FPS, we are good. Eagerly waiting for this.

any updates?

[Luxonis-Brandon]

Thanks for the ping. @saching13 is going to take over integrating this.

[rishabsingh3003]

Any updates on this?

[Luxonis-Brandon]

Oh ya sorry this fell way behind because of other priorities. Sorry about the delay. Technically it can be run, but it's on an old depthai.

Let me check internally.

[rishabsingh3003]

I tested this a little bit, and it seems to be working. The detection isn't very smooth (i.e it detects it and then loses it again and again..) but overall it seems good. Any plans on getting this merged?

[rishabsingh3003]

ping

[Luxonis-Brandon]

Will ask. Not sure what blockers are to merging. April Tags seems to be the one thing we can never merge... :-). Not immediately sure why.

[SzabolcsGergely]

Update:

apriltags.mp4

Algorithm params are configurable to improve accuracy or performance.
https://github.com/luxonis/depthai-python/blob/gen2_apriltag/examples/AprilTag/apriltag.py

[MAVProxyUser]

I tested this a little bit, and it seems to be working. The detection isn't very smooth (i.e it detects it and then loses it again and again..)

This was my experience as well. I was a little disappointed that raw OpenCV + Apriltag library seems to perform way better. luxonis/depthai#133 (comment)

Are there plans moving forward to revamp this code ?

[themarpe]

@MAVProxyUser

No, on RVC2 there are no plans to improve AprilTag, but we'll perhaps do so on RVC3/... onwards

[MAVProxyUser]

That is unfortunate as it feels nearly unusable as is.

[themarpe]

Yes - we'll likely deprecate it for RVC2 in the upcoming releases

[GOBish]

Any updates on apriltags?

[Erol444]

@GOBish as mentioned above, we deprecated it on RVC2 platform due to poor performance (fps/accuracy). It will be available on the upcoming RVC4 platform though

[GOBish]

Saw that, thanks! Will the original cameras be able to support RVC4?

[Erol444]

Hi @GOBish , no, there will be new cameras (OAK4) that will have an integrated RVC4 platform.