Automated framed screenshots #28

mkchoi212 · 2018-04-01T07:13:38Z

Major feat in this PR are

Support for fastlane screenshots with localization
New UI tests for the new shiny UI with localization

I wasn't sure which screens to capture and what kind of text to put in the frames but here's what I came up with. Here's some demo screenshots in German; thanks to Google Translate 😃

Please note that code for screenshooting is part of UI testing

mkchoi212 · 2018-04-01T07:54:01Z

circleci seems to be getting the following error, which I'm not getting on my local machine

❌  /Users/distiller/project/VLC for iOSUITests/VLCiOSTestVideoCodecs.swift:13:8: cannot load underlying module for 'XCTest'
[00:21:20]: ▸ import XCTest

dcordero · 2018-04-01T10:43:12Z

VLC for iOSUITests/VLCiOSTestMenu.swift

+//  - Audio tab home
+//  - Local Network tab home
+//  - Video tab home
+//


This header comment does not follow the format of the project, specially regarding the license part. Please check an existing file as reference.

dcordero · 2018-04-01T10:43:50Z

VLC for iOSUITests/VLCiOSTestMenu.swift

+
+    override func tearDown() {
+        super.tearDown()
+    }


Could this method be removed?

dcordero · 2018-04-01T10:47:02Z

VLC for iOSUITests/VLCiOSTestVideoCodecs.swift

+//
+//  Screenshots taken
+//  - Video playback (Jellyfish)
+//


Same here, this header comment does not follow the project format

dcordero · 2018-04-01T10:48:19Z

VLC for iOSUITests/VLCiOSTestVideoCodecs.swift

+
+    override func tearDown() {
+        super.tearDown()
+    }


Same here, I think this method could be removed

dcordero · 2018-04-01T10:49:08Z

VLC for iOSUITests/XCUIElement+Helpers.swift

+//
+//  Created by Mike Choi on 3/30/18.
+//  Copyright © 2018 VideoLAN. All rights reserved.
+//


☝️ 🤓

dcordero · 2018-04-01T10:51:45Z

Podfile

@@ -23,6 +23,7 @@ def iOS_pods
  pod 'MediaLibraryKit-prod'
  pod 'MobileVLCKit', '3.0.2'
  pod 'GTMAppAuth'
+  pod 'SimulatorStatusMagic', :configurations => ['Debug']


This is adding the dependency to the main application targets increasing their bundle sizes. Could it be possible to define it only in the uitests target?

Moving pod 'SimulatorStatusMagic' to target 'VLC for iOSUITests' won't allow VLCAppDelegate.mto import the module, right?

https://github.com/mkchoi212/vlc-ios/blob/6383eee5f46849f493a4ee29e6b7e9224f2c0622/Sources/VLCAppDelegate.m#L38

correct! @dcordero correct me if I'm wrong here please.
I think instead of importing it directly you have an extension on Appdelegate for the tests and that extension is only part of the testmodule. You would also not need the debug configuration that way. Though I'm not sure how that works with classmethods. I would take a look at Artsys App here

dcordero · 2018-04-01T11:13:38Z

Great work, the screenshots look awesome !!!

I have added a few comments here and there

dcordero · 2018-04-02T09:32:29Z

The error in circleci means that XCTest is been imported from an Application target. Are the those new swift files included in the test target or the app target? That could be the reason of the error.

carolanitz

This looks already soooo great!! I had some inline comments. I can't wait until this is ready to be merged! :D
Btw for screenshotting we got the permission to use the following files :
https://www.youtube.com/watch?v=ba62uuv-5Dc
https://www.youtube.com/watch?v=6NDYf3RZ1eY
I'm not sure how this would work exactly but there is a way to have these videos (after downloading them from youtube)in the User Directory. That way we actually have files in the Video and Audio tab respectively. This should probably be part of an additional PR though :)

carolanitz · 2018-04-02T07:51:11Z

.gitignore

@@ -10,7 +10,6 @@ xcuserdata
 # Fastlane
 fastlane/report.xml
 fastlane/Preview.html
-fastlane/screenshots


According to the fastlane docs it's recommended to not store the screenshots in the repository
https://docs.fastlane.tools/best-practices/source-control/

I will make modifications based on this guide! 👍

awesome that's even better! Great find :)

carolanitz · 2018-04-02T08:05:49Z

Podfile

@@ -23,6 +23,7 @@ def iOS_pods
  pod 'MediaLibraryKit-prod'
  pod 'MobileVLCKit', '3.0.2'
  pod 'GTMAppAuth'
+  pod 'SimulatorStatusMagic', :configurations => ['Debug']


correct! @dcordero correct me if I'm wrong here please.
I think instead of importing it directly you have an extension on Appdelegate for the tests and that extension is only part of the testmodule. You would also not need the debug configuration that way. Though I'm not sure how that works with classmethods. I would take a look at Artsys App here

carolanitz · 2018-04-02T08:07:00Z

VLC for iOSUITests/VLCiOSTestMenu.swift

+
+class VLCiOSTestMenu: XCTestCase {
+    let app = XCUIApplication()
+    let moreTab = XCUIApplication().tabBars.buttons.element(boundBy: 4)


Is it possible to get the tab by localized title instead ?

There is no localizable string for this :(
I think the "More" tab is using Apple's built-in localized strings.

carolanitz · 2018-04-02T08:09:29Z

VLC for iOSUITests/VLCiOSTestMenu.swift

+    }
+
+    func localizedString(key: String) -> String {
+        let localizationBundle = Bundle(path: Bundle(for: VLCiOSTestMenu.self).path(forResource: deviceLanguage, ofType: ".lproj")!)!


force unwrap is not great. How do you feel about making the tests fail if the localization file can't be loaded?

Additionally it should be enough to load the bundle once for every language. That would speed up the test

carolanitz · 2018-04-02T08:11:26Z

VLC for iOSUITests/VLCiOSTestMenu.swift

+    }
+
+    func testNavigationToVideoTab() {
+        app.tabBars.buttons["Video"].tap()


Is there no localizable string ?

carolanitz · 2018-04-02T08:19:07Z

VLC for iOSUITests/VLCiOSTestVideoCodecs.swift

+    func localizedString(key: String) -> String {
+        let localizationBundle = Bundle(path: Bundle(for: VLCiOSTestMenu.self).path(forResource: deviceLanguage, ofType: ".lproj")!)!
+        let result = NSLocalizedString(key, bundle: localizationBundle, comment: "")
+        return result


seeing that this is used here as well it might be even good to have a testhelpers method ? We should definitely not duplicate the code here

carolanitz · 2018-04-02T08:21:27Z

fastlane/Snapfile

@@ -0,0 +1,13 @@
+ devices([
+   "iPhone X",
+   "iPhone 8"


I think we will need all the devices :D

carolanitz · 2018-04-02T08:21:58Z

fastlane/Snapfile

+ languages([
+   "en",
+   "de",
+   "fr",


we will also need all the languages that we support. And yes I know this will take forever

To create the framed screenshots, we need localized keyword.string and title.string for each language. Maybe this should be a progressive effort once we officially decide on what text to put on the framed screenshots??

absolutely let's just ask Louis!

carolanitz · 2018-04-02T08:27:58Z

fastlane/SnapshotHelper.swift

+    }
+}
+
+// Please don't remove the lines below


I'm expecting that you double checked and understood this file btw :)

carolanitz

we should switch to the accessibility element identifiers. that will resolve the bundle and localization issues. Also the simulator magic only as part of the Testtarget and adding it in the setup for the tests should resolve the import conundrum

carolanitz · 2018-04-03T13:48:21Z

VLC for iOSUITests/VLCiOSTestMenu.swift

+        XCUIDevice.shared.orientation = .portrait
+        setupSnapshot(app)
+        helper = TestHelper(lang: deviceLanguage, target: VLCiOSTestMenu.self)
+        app.launch()


This here would actually be a better place for the status magic (but remember to disable it again in tearDown()).

carolanitz · 2018-04-03T13:51:37Z

VLC for iOSUITests/TestHelper.swift

+import XCTest
+
+struct TestHelper {
+    let localizationBundle: Bundle


something that I didn't think about in my earlier review but that a friend pointed out is to use accessibility element identifiers. They don’t need to be localized! Since we need to make the app accessibility friendly anyway this will kill two birds with one stone :)

So it looks like none of the elements in the app has its accessibilityIdentifier set. How should we go about this? Update every single element's accessibilityIdentifier?

yes exactly. It needs to be added

carolanitz · 2018-04-03T13:53:08Z

VLC for iOSUITests/VLCiOSTestMenu.swift

+        helper.tap(tabDescription: audio, app: app)
+        XCTAssertNotNil(app.navigationBars[audio])
+
+        snapshot("audio_tab")


btw could we separate the Unit Tests from the snapshot taking?

carolanitz · 2018-04-04T21:41:03Z

.gitignore

 fastlane/test_output

+fastlane/screenshots/*/*.png
+fastlane/screenshots/screenshots.html


will you hate me if I ask for an alphabetically ordered .gitignore 🙊? At least within the fastlane section

Not at all 😄

carolanitz · 2018-04-09T12:12:14Z

looks already better. Could you use the accessibilityidentifier on the tabbaritems and other views as well? We need to avoid all use of the localization bundle otherwise renaming a tab would lead to failing tests for no reason

carolanitz · 2018-04-09T12:54:08Z

There is also something misconfigured in the xcodeproj for me. Might be just me though

carolanitz · 2018-04-09T13:02:53Z

fastlane/Snapfile

+   "iPad Pro (12.9-inch)", 
+   "iPad Pro (12.9-inch) (2nd generation)", 
+   "iPad Pro (10.5-inch)"
+ ])


carolanitz · 2018-04-09T13:17:54Z

the tests fail on iPad btw but the more tab will disappear anyway soon. And one more question. What are the screenshots doing for a language that is not defined? can we make sure that it uses the screenshots from the English version?

mkchoi212 · 2018-04-09T13:47:12Z

@carolanitz Right now, frameitdoes not work with any language that does not have their respective .string files. But let me see if I can make the default language English for them!

mkchoi212 · 2018-04-12T01:41:00Z

Please note that because the "More" tab is going to be removed soon, the test does not take the "More" tab into consideration. Because of this, the tests fail for all iPhones but succeed for all iPads

carolanitz · 2018-04-13T12:52:10Z

the merge conflicts just need to be resolved and I think this is good to go

mkchoi212 · 2018-04-16T20:53:11Z

Fixed all the merge conflicts!
Sorry for such a huge PR but thanks for the comments @dcordero @carolanitz ❤️
I'll make sure to break up these bad boys into smaller chunks next time!

Perfect status bar displays 100% battery life and 9:41 as the time. This allows for consistent, and perfect screenshoting, 100% of the time

To take framed screenshots, run `fastlane screenshots` Note that the list of supported devices and languages is random for now.

Notice that screenshots for fastlane are taken in the test file by calling `snapshot("IMG_NAME")`

Note that only German, French, and English have been added so far. Design/content for the framed screenshots are BASIC and NEEDS IMPROVEMENT.

iOS test cases are now compatible with both iPhone and iPad.

Note that a new target has been created in the Podfile specifically for iOS UITests.

Fixed bugs regarding change of localization during screenshoting

Note that string literals are being used all over the project to create identifiers. A new file containing all identifier literals should be created in the future.

Fastlane does not support default langauges. Therefore, in order to generate screenshots for all languages, default English keyword.strings and title.strings have been copied for all langauges.

mkchoi212 · 2018-04-18T19:08:07Z

All commits have been squashed!

mkchoi212 · 2018-04-18T19:10:42Z

VLC.xcodeproj/project.pbxproj

@@ -3431,7 +3408,6 @@
 				ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES = YES;
 				CLANG_ANALYZER_NONNULL = YES;
 				CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
-				CLANG_CXX_LANGUAGE_STANDARD = "gnu++14";


@carolanitz, this fixes the misconfigured in the xcodeproj problem that you discovered!

dcordero reviewed Apr 1, 2018

View reviewed changes

mkchoi212 force-pushed the master branch from 6383eee to 13d3bf2 Compare April 1, 2018 18:38

carolanitz requested changes Apr 2, 2018

View reviewed changes

mkchoi212 force-pushed the master branch from 13d3bf2 to 00d621f Compare April 3, 2018 07:05

carolanitz requested changes Apr 3, 2018

View reviewed changes

mkchoi212 force-pushed the master branch 2 times, most recently from d5bb0a6 to 6956146 Compare April 4, 2018 02:24

carolanitz reviewed Apr 4, 2018

View reviewed changes

mkchoi212 force-pushed the master branch 3 times, most recently from 7276bcc to 8978eaf Compare April 7, 2018 01:05

carolanitz reviewed Apr 9, 2018

View reviewed changes

fastlane/Snapfile

"iPad Pro (12.9-inch)",

"iPad Pro (12.9-inch) (2nd generation)",

"iPad Pro (10.5-inch)"

])

Copy link

Member

carolanitz Apr 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

dcordero mentioned this pull request Apr 9, 2018

Add Ruby and Swift linters and PR template #27

Closed

5 tasks

mkchoi212 force-pushed the master branch from 8978eaf to 20996a8 Compare April 12, 2018 01:27

mkchoi212 force-pushed the master branch 4 times, most recently from 83c49b1 to d25f37f Compare April 12, 2018 02:04

carolanitz approved these changes Apr 13, 2018

View reviewed changes

mkchoi212 force-pushed the master branch 2 times, most recently from 7036aee to 5b19e9c Compare April 13, 2018 16:30

mkchoi212 added 9 commits April 18, 2018 13:51

[Simulator] Add perfect status bar

78f7053

Perfect status bar displays 100% battery life and 9:41 as the time. This allows for consistent, and perfect screenshoting, 100% of the time

[fastlane] Enable fastlane screenshots

975ca70

To take framed screenshots, run `fastlane screenshots` Note that the list of supported devices and languages is random for now.

[UI] New tests for the new UI

050b23f

Notice that screenshots for fastlane are taken in the test file by calling `snapshot("IMG_NAME")`

[fastlane] Add frameit keyword/title data

a19889e

Note that only German, French, and English have been added so far. Design/content for the framed screenshots are BASIC and NEEDS IMPROVEMENT.

[UI] Modify iOS test cases for iPad

731460b

iOS test cases are now compatible with both iPhone and iPad.

[Simulator] Relocate perfect status bar

7e6593d

Note that a new target has been created in the Podfile specifically for iOS UITests.

[fastlane] Seperate screenshot code from testing code

773b500

Fixed bugs regarding change of localization during screenshoting

[UI] Use accessibilityIdentifier to interact

227814f

Note that string literals are being used all over the project to create identifiers. A new file containing all identifier literals should be created in the future.

[UI] Add default keyword and title files for all languages.

e62a7fb

Fastlane does not support default langauges. Therefore, in order to generate screenshots for all languages, default English keyword.strings and title.strings have been copied for all langauges.

mkchoi212 force-pushed the master branch from 5b19e9c to e62a7fb Compare April 18, 2018 19:07

mkchoi212 commented Apr 18, 2018

View reviewed changes

vlc-mirrorer merged commit e62a7fb into videolan:master Apr 18, 2018

Automated framed screenshots #28

Automated framed screenshots #28

Conversation

mkchoi212 commented Apr 1, 2018 • edited

mkchoi212 commented Apr 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolanitz Apr 2, 2018 • edited

Choose a reason for hiding this comment

dcordero commented Apr 1, 2018 • edited

dcordero commented Apr 2, 2018

carolanitz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolanitz Apr 2, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkchoi212 Apr 2, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolanitz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolanitz Apr 4, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carolanitz commented Apr 9, 2018

carolanitz commented Apr 9, 2018

Choose a reason for hiding this comment

carolanitz commented Apr 9, 2018

mkchoi212 commented Apr 9, 2018

mkchoi212 commented Apr 12, 2018

carolanitz commented Apr 13, 2018

mkchoi212 commented Apr 16, 2018

mkchoi212 commented Apr 18, 2018

Choose a reason for hiding this comment

mkchoi212 commented Apr 1, 2018 •

edited

carolanitz Apr 2, 2018 •

edited

dcordero commented Apr 1, 2018 •

edited

carolanitz Apr 2, 2018 •

edited

mkchoi212 Apr 2, 2018 •

edited

carolanitz Apr 4, 2018 •

edited