[iOS] Transcode videos selected from UIImagePickerController

https://bugs.webkit.org/show_bug.cgi?id=230639
rdar://79665678

Reviewed by Tim Horton.

Source/WebCore:

* en.lproj/Localizable.strings:

Add a localizable string for the message displayed while transcoding
video.

Source/WebCore/PAL:

Add AVFoundation API needed to transcode video.

* pal/cocoa/AVFoundationSoftLink.h:
* pal/cocoa/AVFoundationSoftLink.mm:

Source/WebKit:

File inputs on iOS allow users to choose images/videos from the system
photo picker, using UIImagePickerController. In single selection mode,
UIImagePickerController transcodes the selected video to H.264. However,
in multiple selection mode, video is not transcoded and is left in its
original format.

Today, videos on most iOS devices are encoded with HEVC by default.
However, some sites, such as Twitter, only accept H.264 encoded video.
Thus, the current video upload behavior is problematic, as users may be
unable to upload video.

Unfortunately, the photo picking functionality of UIImagePickerController
is deprecated. The best solution would be to adopt PHPickerViewController,
the replacement API, which performs transcoding when retrieving selected
items (regardless of single/multiple selection). However,
PHPickerViewController currently lacks other functionality that WebKit
requires, preventing adoption.

Consequently, the short term solution is to transcode the videos in
WebKit, ensuring H.264 encoded video is always provided to sites. See
below for implementation details.

* Platform/spi/ios/PhotosUISPI.h: Added.
* UIProcess/ios/forms/WKFileUploadPanel.mm:
(-[_WKFileUploadItem setFileURL:]):

Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.

(-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):

Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
and the display of progress UI. Transcoding is performed serially
(one video at a time), but occurs off the main thread.

(-[WKFileUploadMediaTranscoder start]):

Begin transcoding. Run a timer to update the progress UI, as
AVAssetExportSession does not provide progress updates on its own.

The progress UI is implemented using PUActivityProgressController, to
match system Photos UI.

(-[WKFileUploadMediaTranscoder _processItemAtIndex:]):

Transcode a single video, using AVAssetExportSession. If transcoding
fails for any reason, the original video is used as a fallback.
Transcoding can also be cancelled using the progress UI, in which case
no more videos are processed.

(-[WKFileUploadMediaTranscoder _finishedProcessing]):
(-[WKFileUploadMediaTranscoder _dismissProgress]):
(-[WKFileUploadMediaTranscoder _updateProgress:]):
(-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
(-[WKFileUploadPanel _chooseMediaItems:]):

Refactor the common aspects of single/multiple media selection into a
single method.

(-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
(-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
(-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):

Remove redundant platform conditional.

(-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):

If any videos were selected, transcode them prior to uploading.

* WebKit.xcodeproj/project.pbxproj:


Canonical link: https://commits.webkit.org/242548@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@283592 268f45cc-cd09-0410-ab3c-d52691b4dbfc

Source/WebCore/ChangeLog

@@ -1,3 +1,16 @@
+2021-10-05  Aditya Keerthi  <akeerthi@apple.com>
+
+        [iOS] Transcode videos selected from UIImagePickerController
+        https://bugs.webkit.org/show_bug.cgi?id=230639
+        rdar://79665678
+
+        Reviewed by Tim Horton.
+
+        * en.lproj/Localizable.strings:
+
+        Add a localizable string for the message displayed while transcoding
+        video.
+
 2021-10-05  Chris Dumez  <cdumez@apple.com>
 
         ASSERT(m_callback->hasCallback()) under IntersectionObserver::notify()

Source/WebCore/PAL/ChangeLog

@@ -1,3 +1,16 @@
+2021-10-05  Aditya Keerthi  <akeerthi@apple.com>
+
+        [iOS] Transcode videos selected from UIImagePickerController
+        https://bugs.webkit.org/show_bug.cgi?id=230639
+        rdar://79665678
+
+        Reviewed by Tim Horton.
+
+        Add AVFoundation API needed to transcode video.
+
+        * pal/cocoa/AVFoundationSoftLink.h:
+        * pal/cocoa/AVFoundationSoftLink.mm:
+
 2021-10-05  Ayumi Kojima  <ayumi_kojima@apple.com>
 
         Unreviewed, reverting r283339.

Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.h

@@ -38,6 +38,7 @@ SOFT_LINK_FRAMEWORK_FOR_HEADER(PAL, AVFoundation)
 
 SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetCache)
 SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetCollection)
+SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetExportSession)
 SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetImageGenerator)
 SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetReader)
 SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetReaderSampleReferenceOutput)
@@ -190,6 +191,8 @@ SOFT_LINK_CONSTANT_MAY_FAIL_FOR_HEADER(PAL, AVFoundation, AVVideoCodecTypeHEVCWi
 #define AVVideoCodecTypeHEVCWithAlpha PAL::get_AVFoundation_AVVideoCodecTypeHEVCWithAlpha()
 SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVFileTypeMPEG4, NSString *)
 #define AVFileTypeMPEG4 PAL::get_AVFoundation_AVFileTypeMPEG4()
+SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVFileTypeQuickTimeMovie, NSString *)
+#define AVFileTypeQuickTimeMovie PAL::get_AVFoundation_AVFileTypeQuickTimeMovie()
 SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVVideoCodecKey, NSString *)
 #define AVVideoCodecKey PAL::get_AVFoundation_AVVideoCodecKey()
 SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVVideoCodecH264, NSString *)
@@ -336,4 +339,7 @@ SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioCompressedBuffer)
 SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioPCMBuffer)
 #endif // PLATFORM(COCOA)
 
+SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVAssetExportPresetHighestQuality, NSString *)
+#define AVAssetExportPresetHighestQuality PAL::get_AVFoundation_AVAssetExportPresetHighestQuality()
+
 #endif // USE(AVFOUNDATION)

Source/WebCore/PAL/pal/cocoa/AVFoundationSoftLink.mm

@@ -63,6 +63,7 @@ static BOOL justReturnsNO()
 #endif
 
 SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetCache, PAL_EXPORT)
+SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetExportSession, PAL_EXPORT)
 SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetImageGenerator, PAL_EXPORT)
 SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetReader, PAL_EXPORT)
 SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetWriter, PAL_EXPORT)
@@ -133,6 +134,7 @@ static BOOL justReturnsNO()
 SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVCaptureDeviceWasConnectedNotification, NSString *, PAL_EXPORT)
 SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVCaptureDeviceWasDisconnectedNotification, NSString *, PAL_EXPORT)
 SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVFileTypeMPEG4, NSString *, PAL_EXPORT)
+SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVFileTypeQuickTimeMovie, NSString *, PAL_EXPORT)
 SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResize, NSString *, PAL_EXPORT)
 SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResizeAspect, NSString *, PAL_EXPORT)
 SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResizeAspectFill, NSString *, PAL_EXPORT)
@@ -253,4 +255,6 @@ static BOOL justReturnsNO()
 SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAudioPCMBuffer, PAL_EXPORT)
 #endif
 
+SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetExportPresetHighestQuality, NSString *, PAL_EXPORT)
+
 #endif // USE(AVFOUNDATION)

Source/WebCore/en.lproj/Localizable.strings

@@ -754,6 +754,9 @@
 /* Description of the PostScript type supported by the PDF pseudo plug-in. Visible in the Installed Plug-ins page in Safari. */
 "PostScript" = "PostScript";
 
+/* Title for file upload progress view */
+"Preparing (file upload)" = "Preparing…";
+
 /* Title for Quick Look action button */
 "Quick Look" = "Quick Look";
 

Source/WebKit/ChangeLog

@@ -1,3 +1,83 @@
+2021-10-05  Aditya Keerthi  <akeerthi@apple.com>
+
+        [iOS] Transcode videos selected from UIImagePickerController
+        https://bugs.webkit.org/show_bug.cgi?id=230639
+        rdar://79665678
+
+        Reviewed by Tim Horton.
+
+        File inputs on iOS allow users to choose images/videos from the system
+        photo picker, using UIImagePickerController. In single selection mode,
+        UIImagePickerController transcodes the selected video to H.264. However,
+        in multiple selection mode, video is not transcoded and is left in its
+        original format.
+
+        Today, videos on most iOS devices are encoded with HEVC by default.
+        However, some sites, such as Twitter, only accept H.264 encoded video.
+        Thus, the current video upload behavior is problematic, as users may be
+        unable to upload video.
+
+        Unfortunately, the photo picking functionality of UIImagePickerController
+        is deprecated. The best solution would be to adopt PHPickerViewController,
+        the replacement API, which performs transcoding when retrieving selected
+        items (regardless of single/multiple selection). However,
+        PHPickerViewController currently lacks other functionality that WebKit
+        requires, preventing adoption.
+
+        Consequently, the short term solution is to transcode the videos in
+        WebKit, ensuring H.264 encoded video is always provided to sites. See
+        below for implementation details.
+
+        * Platform/spi/ios/PhotosUISPI.h: Added.
+        * UIProcess/ios/forms/WKFileUploadPanel.mm:
+        (-[_WKFileUploadItem setFileURL:]):
+
+        Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.
+
+        (-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):
+
+        Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
+        and the display of progress UI. Transcoding is performed serially
+        (one video at a time), but occurs off the main thread.
+
+        (-[WKFileUploadMediaTranscoder start]):
+
+        Begin transcoding. Run a timer to update the progress UI, as
+        AVAssetExportSession does not provide progress updates on its own.
+
+        The progress UI is implemented using PUActivityProgressController, to
+        match system Photos UI.
+
+        (-[WKFileUploadMediaTranscoder _processItemAtIndex:]):
+
+        Transcode a single video, using AVAssetExportSession. If transcoding
+        fails for any reason, the original video is used as a fallback.
+        Transcoding can also be cancelled using the progress UI, in which case
+        no more videos are processed.
+
+        (-[WKFileUploadMediaTranscoder _finishedProcessing]):
+        (-[WKFileUploadMediaTranscoder _dismissProgress]):
+        (-[WKFileUploadMediaTranscoder _updateProgress:]):
+        (-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
+        (-[WKFileUploadPanel _chooseMediaItems:]):
+
+        Refactor the common aspects of single/multiple media selection into a
+        single method.
+
+        (-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
+        (-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
+        (-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
+        (-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
+        (-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):
+
+        Remove redundant platform conditional.
+
+        (-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):
+
+        If any videos were selected, transcode them prior to uploading.
+
+        * WebKit.xcodeproj/project.pbxproj:
+
 2021-10-05  Alex Christensen  <achristensen@webkit.org>
 
         Add an entitlement check to only allow AdAttributionDaemon to be connected to by the network process

Source/WebKit/Platform/spi/ios/PhotosUISPI.h

@@ -0,0 +1,49 @@
+/*
+ * Copyright (C) 2021 Apple Inc. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+ * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
+ * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
+ * THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#if USE(APPLE_INTERNAL_SDK)
+
+#import <PhotosUI/PUActivityProgressController.h>
+
+#else
+
+#import "UIKitSPI.h"
+
+@interface PUActivityProgressController : NSObject
+
+@property (nonatomic, copy) NSString *title;
+
+@property (nonatomic, copy) void (^cancellationHandler)(void);
+
+@property (nonatomic, readonly) BOOL isCancelled;
+
+- (void)setFractionCompleted:(double)fractionCompleted;
+
+- (void)showAnimated:(BOOL)animated allowDelay:(BOOL)allowDelay;
+- (void)hideAnimated:(BOOL)animated allowDelay:(BOOL)allowDelay;
+
+@end
+
+#endif // USE(APPLE_INTERNAL_SDK)