Skip to content
Permalink
Browse files
[iOS] Transcode videos selected from UIImagePickerController
https://bugs.webkit.org/show_bug.cgi?id=230639
rdar://79665678

Reviewed by Tim Horton.

Source/WebCore:

* en.lproj/Localizable.strings:

Add a localizable string for the message displayed while transcoding
video.

Source/WebCore/PAL:

Add AVFoundation API needed to transcode video.

* pal/cocoa/AVFoundationSoftLink.h:
* pal/cocoa/AVFoundationSoftLink.mm:

Source/WebKit:

File inputs on iOS allow users to choose images/videos from the system
photo picker, using UIImagePickerController. In single selection mode,
UIImagePickerController transcodes the selected video to H.264. However,
in multiple selection mode, video is not transcoded and is left in its
original format.

Today, videos on most iOS devices are encoded with HEVC by default.
However, some sites, such as Twitter, only accept H.264 encoded video.
Thus, the current video upload behavior is problematic, as users may be
unable to upload video.

Unfortunately, the photo picking functionality of UIImagePickerController
is deprecated. The best solution would be to adopt PHPickerViewController,
the replacement API, which performs transcoding when retrieving selected
items (regardless of single/multiple selection). However,
PHPickerViewController currently lacks other functionality that WebKit
requires, preventing adoption.

Consequently, the short term solution is to transcode the videos in
WebKit, ensuring H.264 encoded video is always provided to sites. See
below for implementation details.

* Platform/spi/ios/PhotosUISPI.h: Added.
* UIProcess/ios/forms/WKFileUploadPanel.mm:
(-[_WKFileUploadItem setFileURL:]):

Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.

(-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):

Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
and the display of progress UI. Transcoding is performed serially
(one video at a time), but occurs off the main thread.

(-[WKFileUploadMediaTranscoder start]):

Begin transcoding. Run a timer to update the progress UI, as
AVAssetExportSession does not provide progress updates on its own.

The progress UI is implemented using PUActivityProgressController, to
match system Photos UI.

(-[WKFileUploadMediaTranscoder _processItemAtIndex:]):

Transcode a single video, using AVAssetExportSession. If transcoding
fails for any reason, the original video is used as a fallback.
Transcoding can also be cancelled using the progress UI, in which case
no more videos are processed.

(-[WKFileUploadMediaTranscoder _finishedProcessing]):
(-[WKFileUploadMediaTranscoder _dismissProgress]):
(-[WKFileUploadMediaTranscoder _updateProgress:]):
(-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
(-[WKFileUploadPanel _chooseMediaItems:]):

Refactor the common aspects of single/multiple media selection into a
single method.

(-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
(-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
(-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):

Remove redundant platform conditional.

(-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):

If any videos were selected, transcode them prior to uploading.

* WebKit.xcodeproj/project.pbxproj:


Canonical link: https://commits.webkit.org/242548@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@283592 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
pxlcoder committed Oct 6, 2021
1 parent 63b0232 commit 61792f44434dd130448a27ac0cc6076f7554cf52
Showing 9 changed files with 404 additions and 38 deletions.
@@ -1,3 +1,16 @@
2021-10-05 Aditya Keerthi <akeerthi@apple.com>

[iOS] Transcode videos selected from UIImagePickerController
https://bugs.webkit.org/show_bug.cgi?id=230639
rdar://79665678

Reviewed by Tim Horton.

* en.lproj/Localizable.strings:

Add a localizable string for the message displayed while transcoding
video.

2021-10-05 Chris Dumez <cdumez@apple.com>

ASSERT(m_callback->hasCallback()) under IntersectionObserver::notify()
@@ -1,3 +1,16 @@
2021-10-05 Aditya Keerthi <akeerthi@apple.com>

[iOS] Transcode videos selected from UIImagePickerController
https://bugs.webkit.org/show_bug.cgi?id=230639
rdar://79665678

Reviewed by Tim Horton.

Add AVFoundation API needed to transcode video.

* pal/cocoa/AVFoundationSoftLink.h:
* pal/cocoa/AVFoundationSoftLink.mm:

2021-10-05 Ayumi Kojima <ayumi_kojima@apple.com>

Unreviewed, reverting r283339.
@@ -38,6 +38,7 @@ SOFT_LINK_FRAMEWORK_FOR_HEADER(PAL, AVFoundation)

SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetCache)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetCollection)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetExportSession)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetImageGenerator)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetReader)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAssetReaderSampleReferenceOutput)
@@ -190,6 +191,8 @@ SOFT_LINK_CONSTANT_MAY_FAIL_FOR_HEADER(PAL, AVFoundation, AVVideoCodecTypeHEVCWi
#define AVVideoCodecTypeHEVCWithAlpha PAL::get_AVFoundation_AVVideoCodecTypeHEVCWithAlpha()
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVFileTypeMPEG4, NSString *)
#define AVFileTypeMPEG4 PAL::get_AVFoundation_AVFileTypeMPEG4()
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVFileTypeQuickTimeMovie, NSString *)
#define AVFileTypeQuickTimeMovie PAL::get_AVFoundation_AVFileTypeQuickTimeMovie()
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVVideoCodecKey, NSString *)
#define AVVideoCodecKey PAL::get_AVFoundation_AVVideoCodecKey()
SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVVideoCodecH264, NSString *)
@@ -336,4 +339,7 @@ SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioCompressedBuffer)
SOFT_LINK_CLASS_FOR_HEADER(PAL, AVAudioPCMBuffer)
#endif // PLATFORM(COCOA)

SOFT_LINK_CONSTANT_FOR_HEADER(PAL, AVFoundation, AVAssetExportPresetHighestQuality, NSString *)
#define AVAssetExportPresetHighestQuality PAL::get_AVFoundation_AVAssetExportPresetHighestQuality()

#endif // USE(AVFOUNDATION)
@@ -63,6 +63,7 @@ static BOOL justReturnsNO()
#endif

SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetCache, PAL_EXPORT)
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetExportSession, PAL_EXPORT)
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetImageGenerator, PAL_EXPORT)
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetReader, PAL_EXPORT)
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetWriter, PAL_EXPORT)
@@ -133,6 +134,7 @@ static BOOL justReturnsNO()
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVCaptureDeviceWasConnectedNotification, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVCaptureDeviceWasDisconnectedNotification, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVFileTypeMPEG4, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVFileTypeQuickTimeMovie, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResize, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResizeAspect, NSString *, PAL_EXPORT)
SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVLayerVideoGravityResizeAspectFill, NSString *, PAL_EXPORT)
@@ -253,4 +255,6 @@ static BOOL justReturnsNO()
SOFT_LINK_CLASS_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAudioPCMBuffer, PAL_EXPORT)
#endif

SOFT_LINK_CONSTANT_FOR_SOURCE_WITH_EXPORT(PAL, AVFoundation, AVAssetExportPresetHighestQuality, NSString *, PAL_EXPORT)

#endif // USE(AVFOUNDATION)
@@ -754,6 +754,9 @@
/* Description of the PostScript type supported by the PDF pseudo plug-in. Visible in the Installed Plug-ins page in Safari. */
"PostScript" = "PostScript";

/* Title for file upload progress view */
"Preparing (file upload)" = "Preparing…";

/* Title for Quick Look action button */
"Quick Look" = "Quick Look";

@@ -1,3 +1,83 @@
2021-10-05 Aditya Keerthi <akeerthi@apple.com>

[iOS] Transcode videos selected from UIImagePickerController
https://bugs.webkit.org/show_bug.cgi?id=230639
rdar://79665678

Reviewed by Tim Horton.

File inputs on iOS allow users to choose images/videos from the system
photo picker, using UIImagePickerController. In single selection mode,
UIImagePickerController transcodes the selected video to H.264. However,
in multiple selection mode, video is not transcoded and is left in its
original format.

Today, videos on most iOS devices are encoded with HEVC by default.
However, some sites, such as Twitter, only accept H.264 encoded video.
Thus, the current video upload behavior is problematic, as users may be
unable to upload video.

Unfortunately, the photo picking functionality of UIImagePickerController
is deprecated. The best solution would be to adopt PHPickerViewController,
the replacement API, which performs transcoding when retrieving selected
items (regardless of single/multiple selection). However,
PHPickerViewController currently lacks other functionality that WebKit
requires, preventing adoption.

Consequently, the short term solution is to transcode the videos in
WebKit, ensuring H.264 encoded video is always provided to sites. See
below for implementation details.

* Platform/spi/ios/PhotosUISPI.h: Added.
* UIProcess/ios/forms/WKFileUploadPanel.mm:
(-[_WKFileUploadItem setFileURL:]):

Add a setter to update the file URL. Called after transcoding a _WKFileUploadItem.

(-[WKFileUploadMediaTranscoder initWithItems:videoCount:completionHandler:]):

Introduce WKFileUploadMediaTranscoder to manage transcoding of videos
and the display of progress UI. Transcoding is performed serially
(one video at a time), but occurs off the main thread.

(-[WKFileUploadMediaTranscoder start]):

Begin transcoding. Run a timer to update the progress UI, as
AVAssetExportSession does not provide progress updates on its own.

The progress UI is implemented using PUActivityProgressController, to
match system Photos UI.

(-[WKFileUploadMediaTranscoder _processItemAtIndex:]):

Transcode a single video, using AVAssetExportSession. If transcoding
fails for any reason, the original video is used as a fallback.
Transcoding can also be cancelled using the progress UI, in which case
no more videos are processed.

(-[WKFileUploadMediaTranscoder _finishedProcessing]):
(-[WKFileUploadMediaTranscoder _dismissProgress]):
(-[WKFileUploadMediaTranscoder _updateProgress:]):
(-[WKFileUploadMediaTranscoder _temporaryDirectoryCreateIfNecessary]):
(-[WKFileUploadPanel _chooseMediaItems:]):

Refactor the common aspects of single/multiple media selection into a
single method.

(-[WKFileUploadPanel imagePickerController:didFinishPickingMediaWithInfo:]):
(-[WKFileUploadPanel imagePickerController:didFinishPickingMultipleMediaWithInfo:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:successBlock:failureBlock:]):
(-[WKFileUploadPanel _processMediaInfoDictionaries:atIndex:processedResults:successBlock:failureBlock:]):
(-[WKFileUploadPanel _uploadItemFromMediaInfo:successBlock:failureBlock:]):

Remove redundant platform conditional.

(-[WKFileUploadPanel _uploadMediaItemsTranscodingVideo:]):

If any videos were selected, transcode them prior to uploading.

* WebKit.xcodeproj/project.pbxproj:

2021-10-05 Alex Christensen <achristensen@webkit.org>

Add an entitlement check to only allow AdAttributionDaemon to be connected to by the network process
@@ -0,0 +1,49 @@
/*
* Copyright (C) 2021 Apple Inc. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY APPLE INC. AND ITS CONTRIBUTORS ``AS IS''
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
* THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE INC. OR ITS CONTRIBUTORS
* BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
* THE POSSIBILITY OF SUCH DAMAGE.
*/

#if USE(APPLE_INTERNAL_SDK)

#import <PhotosUI/PUActivityProgressController.h>

#else

#import "UIKitSPI.h"

@interface PUActivityProgressController : NSObject

@property (nonatomic, copy) NSString *title;

@property (nonatomic, copy) void (^cancellationHandler)(void);

@property (nonatomic, readonly) BOOL isCancelled;

- (void)setFractionCompleted:(double)fractionCompleted;

- (void)showAnimated:(BOOL)animated allowDelay:(BOOL)allowDelay;
- (void)hideAnimated:(BOOL)animated allowDelay:(BOOL)allowDelay;

@end

#endif // USE(APPLE_INTERNAL_SDK)

0 comments on commit 61792f4

Please sign in to comment.