iOS camera provider enhancements #7857

RobertFlatt · 2022-04-04T02:26:12Z

iOS camera provider enhancements, these are iOS only but implemented in the camera provider shared by iOS and MacOS.

The additional apis implement basic mobile camera functionality, this is not currently implemented.

To provide basic camera services on iOS, this PR adds

 def get_device_orientation(self):

 def change_camera_input(self, index):

 def zoom_level(self, level):

 def get_app_documents_directory(self):

 def save_texture(self, texture, filepath = '', quality = 0.9):
   # saves to iOS Photos App if filepath not specified,
   # or app documents directory if filepath is specified.

The fork also builds and runs on MacOS, indicating it is unlikely the changes have broken the MacOS camera provider.

TODO, but not part of this PR:

1) kivy-ios does not load the camera provider, the Camera widget fails. Assuming this PR is accepted in a timely fashion, I will implement kivy-ios PR that fixes the load and references this enhanced camera provider.

iOS has native photo and video capture apis, these are not currently implemented in this camera provider.

welcome · 2022-04-04T02:26:13Z

Thanks for opening your first pull request here! 💖 Please check out our contributing guidelines.

RobertFlatt · 2022-04-05T02:56:49Z

kivy-ios does not load the camera provider, the Camera widget fails. Assuming this PR is accepted in a timely fashion, I will implement kivy-ios PR that fixes the load and references this enhanced camera provider.

Not required with kivy-ios master.

misl6

Code LGTM at a first glance!

I will test it on runtime (and will try to write a demo App, unless you already have one to share) during the upcoming weekend 😃

Super-minor: How about adding some logging when functionalities are not expected to be available but are called on macOS or vice-versa? Comments are super-fine, but users tend to not read them immediately. 😆

RobertFlatt · 2022-04-07T21:45:32Z

This example uses the new features https://github.com/Android-for-Python/INDEX-of-Examples#c4k_photo_example

It depends on the Preview widget not the Camera widget. For iOS only, the master version of Preview is required https://github.com/Android-for-Python/Camera4Kivy . Release waits on this PR and a kivy-ios that points to the new Kivy master.

The readme has not been updated for ios. Example shows switch between cameras, capture photo or screenshot, pinch-spread for zoom (on my SE only the back camera), device rotation behavior, mirrored front camera.

To enable use of the Camera

        <key>NSCameraUsageDescription</key>
	<string> </string>

To enable save to Photos App

	<key>NSPhotoLibraryAddUsageDescription</key>
	<string> </string>

To make contents of local storage visible to File Manager

	<key>UIFileSharingEnabled</key>
	<true/>
        <key>LSSupportsOpeningDocumentsInPlace</key>
	<true/>

Image saves default to Photos App. The Preview widget storage selection is the same as for Android, so to save to app storage in place of Photos App capture_photo(location = 'private') https://github.com/Android-for-Python/Camera4Kivy#location

[I started this because I wanted to try to port the tflite recipe, but my test example used a camera, so ...]

misl6

Tested it on my iPhone 12 Pro, and generally works good!

Unfortunately, after some minor edits to your example and Camera4Kivy, I wasn't able to attach the ultra-wide or telephoto (back) lenses due to how camera selection is handled.

I've added some comments explaining what could be done to improve it.

misl6 · 2022-04-09T15:46:19Z

kivy/core/camera/camera_avfoundation_implem.m

+    NSArray *devices;
+    AVCaptureDevice *device;
+
+    devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];


devicesWithMediaType has been deprecated in iOS 10.0, iPadOS 10.0, macOS 10.15, Mac Catalyst 13.1 and it doesn't provide access to ultra-wide or telephoto camera lenses.

Instead,AVCaptureDeviceDiscoverySession should be used.

misl6 · 2022-04-09T16:00:07Z

kivy/core/camera/camera_avfoundation_implem.m

+    if (_cameraNum >= 0) {
+        int camNum = _cameraNum % [devices count];
+        if (camNum != _cameraNum) {
+            NSLog(@"Warning: Max Camera Num is %lu; Using camera %d\n",
+                  (unsigned long)([devices count] - 1), camNum);
+        }
+        device = [devices objectAtIndex:camNum];
+        cameraNum = _cameraNum;
+    } else {
+        device = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]  ;
+        cameraNum = 0;
+    }


The new discoverySessionWithDeviceTypes:mediaType:position: allows a fairly more granular selection of the camera.

The devices array may be not consistent of returning the "correct" order of the cameras, so could be nice to create an enumerator on Kivy side, and pass the configuration requested from the user to discoverySessionWithDeviceTypes:mediaType:position:

RobertFlatt · 2022-04-10T02:31:57Z

The api mentioned depends on ios10+ , (last time I looked) we build with minimum 9 not 10. And this is not adjustable __IPHONE_OS_VERSION_MIN_REQUIRED fixed for toolchain build kivy-ios#691 On the subject of min api, the photo capture api gets good with ios11.
I don't understand enough about deviceTypes https://developer.apple.com/documentation/avfoundation/avcapturedevicetype or Apple devices to implement this. It is not clear to me what the options do, or any side effects.
I expect it is possible to synthesize some useful abstraction that extends Kivy's camera = [0,1] but at this point I don't know what it should be, or even what the usage model would be. So I would not try.
This PR provides basic iOS functionality, which we don't have without the PR. We already know the camera provider could be improved, see the TODO at the top of the file. Support for ultrawide and telephoto should be added to that list. Approve this PR as a starting point. Anybody who understands the additional enhancement design issues can then jump in.

misl6 · 2022-04-10T09:00:40Z

Yeah, the API mentioned depends on iOS 10+, and we're still targeting iOS 9 minimum. (Maybe is a good idea to higher that requirement?). But, considering that on iOS is always a good practice to check for the availability of APIs (or for the iOS version that introduced that feature), we can easily switch between the two implementations during runtime.
Decrypting Apple's docs is always a nightmare, so not your fault 😄
I agree that this PR is an improvement to the current situation, and will merge it as-is if you don't expect to continue working on it in the near/mid-term. In that case, I will open an issue that we'll need to likely target before releasing 2.2.x regarding the deprecated code and missing support for non-standard cameras

RobertFlatt · 2022-04-11T01:39:25Z

Updated the comments at the top.
I have other priorities so this is as far as I will take it.
In the future maybe further enhancements will pop to the top of my stack.

RobertFlatt · 2022-04-20T06:16:00Z

@misl6 Can you approve this, please.

misl6

LGTM. Thank you!
I will open a couple of issues targeting what we discussed during the review.

welcome · 2022-04-20T17:17:09Z

Congrats on merging your first pull request! 🎉🎉🎉

RobertFlatt · 2022-04-21T02:16:59Z

@misl6 You open to a kivy-ios PR that bumps the kivy version to include this change?

https://github.com/RobertFlatt/kivy-ios/blob/master/kivy_ios/recipes/kivy/__init__.py#L10

misl6 · 2022-04-23T07:45:45Z

LGTM. Thank you!
I will open a couple of issues targeting what we discussed during the review.

See:
#7871

RobertFlatt added 11 commits April 2, 2022 17:51

Update camera_avfoundation.pyx

4745e03

Update camera_avfoundation_implem.h

9c2c244

Update camera_avfoundation_implem.m

34a3943

Update camera_avfoundation_implem.m

608664e

Update camera_avfoundation_implem.m

3a88c7f

Update camera_avfoundation_implem.h

a9335a5

Update camera_avfoundation.pyx

19f2fab

Update camera_avfoundation_implem.m

1101a5e

Update camera_avfoundation_implem.h

ecb03b4

Update camera_avfoundation.pyx

6ec515d

Update camera_avfoundation_implem.m

d4fa24c

misl6 reviewed Apr 7, 2022

View reviewed changes

misl6 requested changes Apr 9, 2022

View reviewed changes

Update camera_avfoundation_implem.m

ec62b04

misl6 approved these changes Apr 20, 2022

View reviewed changes

misl6 merged commit 0e77e2a into kivy:master Apr 20, 2022

misl6 added this to the 2.2.0 milestone Apr 21, 2022

misl6 added the Component: core-providers kivy/core label Apr 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

iOS camera provider enhancements #7857

iOS camera provider enhancements #7857

RobertFlatt commented Apr 4, 2022 •

edited

welcome bot commented Apr 4, 2022

RobertFlatt commented Apr 5, 2022

misl6 left a comment

RobertFlatt commented Apr 7, 2022

misl6 left a comment

misl6 Apr 9, 2022

misl6 Apr 9, 2022

RobertFlatt commented Apr 10, 2022

misl6 commented Apr 10, 2022

RobertFlatt commented Apr 11, 2022

RobertFlatt commented Apr 20, 2022

misl6 left a comment

welcome bot commented Apr 20, 2022

RobertFlatt commented Apr 21, 2022

misl6 commented Apr 23, 2022

iOS camera provider enhancements #7857

iOS camera provider enhancements #7857

Conversation

RobertFlatt commented Apr 4, 2022 • edited

welcome bot commented Apr 4, 2022

RobertFlatt commented Apr 5, 2022

misl6 left a comment

Choose a reason for hiding this comment

RobertFlatt commented Apr 7, 2022

misl6 left a comment

Choose a reason for hiding this comment

misl6 Apr 9, 2022

Choose a reason for hiding this comment

misl6 Apr 9, 2022

Choose a reason for hiding this comment

RobertFlatt commented Apr 10, 2022

misl6 commented Apr 10, 2022

RobertFlatt commented Apr 11, 2022

RobertFlatt commented Apr 20, 2022

misl6 left a comment

Choose a reason for hiding this comment

welcome bot commented Apr 20, 2022

RobertFlatt commented Apr 21, 2022

misl6 commented Apr 23, 2022

RobertFlatt commented Apr 4, 2022 •

edited