Automated framed screenshots #28

mkchoi212 · 2018-04-01T07:13:38Z

Major feat in this PR are

Support for fastlane screenshots with localization
New UI tests for the new shiny UI with localization

I wasn't sure which screens to capture and what kind of text to put in the frames but here's what I came up with. Here's some demo screenshots in German; thanks to Google Translate 😃

Please note that code for screenshooting is part of UI testing

mkchoi212 · 2018-04-01T07:54:01Z

circleci seems to be getting the following error, which I'm not getting on my local machine

❌  /Users/distiller/project/VLC for iOSUITests/VLCiOSTestVideoCodecs.swift:13:8: cannot load underlying module for 'XCTest'
[00:21:20]: ▸ import XCTest

dcordero · 2018-04-01T11:06:02Z

Sign in to view

+//  - Audio tab home
+//  - Local Network tab home
+//  - Video tab home
+//


This header comment does not follow the format of the project, specially regarding the license part. Please check an existing file as reference.

dcordero · 2018-04-01T11:06:02Z

Sign in to view

+
+    override func tearDown() {
+        super.tearDown()
+    }


Could this method be removed?

dcordero · 2018-04-01T11:06:02Z

Sign in to view

+//
+//  Screenshots taken
+//  - Video playback (Jellyfish)
+//


Same here, this header comment does not follow the project format

dcordero · 2018-04-01T11:06:02Z

Sign in to view

+
+    override func tearDown() {
+        super.tearDown()
+    }


Same here, I think this method could be removed

dcordero · 2018-04-01T11:06:02Z

Sign in to view

+//
+//  Created by Mike Choi on 3/30/18.
+//  Copyright © 2018 VideoLAN. All rights reserved.
+//


☝️ 🤓

dcordero · 2018-04-01T11:06:02Z

Sign in to view

@@ -23,6 +23,7 @@ def iOS_pods
  pod 'MediaLibraryKit-prod'
  pod 'MobileVLCKit', '3.0.2'
  pod 'GTMAppAuth'
+  pod 'SimulatorStatusMagic', :configurations => ['Debug']


This is adding the dependency to the main application targets increasing their bundle sizes. Could it be possible to define it only in the uitests target?

Moving pod 'SimulatorStatusMagic' to target 'VLC for iOSUITests' won't allow VLCAppDelegate.mto import the module, right?

https://github.com/mkchoi212/vlc-ios/blob/6383eee5f46849f493a4ee29e6b7e9224f2c0622/Sources/VLCAppDelegate.m#L38

correct! @dcordero correct me if I'm wrong here please.
I think instead of importing it directly you have an extension on Appdelegate for the tests and that extension is only part of the testmodule. You would also not need the debug configuration that way. Though I'm not sure how that works with classmethods. I would take a look at Artsys App here

dcordero · 2018-04-01T11:13:38Z

Great work, the screenshots look awesome !!!

I have added a few comments here and there

dcordero · 2018-04-02T09:32:29Z

The error in circleci means that XCTest is been imported from an Application target. Are the those new swift files included in the test target or the app target? That could be the reason of the error.

carolanitz

This looks already soooo great!! I had some inline comments. I can't wait until this is ready to be merged! :D
Btw for screenshotting we got the permission to use the following files :
https://www.youtube.com/watch?v=ba62uuv-5Dc
https://www.youtube.com/watch?v=6NDYf3RZ1eY
I'm not sure how this would work exactly but there is a way to have these videos (after downloading them from youtube)in the User Directory. That way we actually have files in the Video and Audio tab respectively. This should probably be part of an additional PR though :)

carolanitz · 2018-04-02T10:08:45Z

Sign in to view

@@ -10,7 +10,6 @@ xcuserdata
 # Fastlane
 fastlane/report.xml
 fastlane/Preview.html
-fastlane/screenshots


According to the fastlane docs it's recommended to not store the screenshots in the repository
https://docs.fastlane.tools/best-practices/source-control/

I will make modifications based on this guide! 👍

awesome that's even better! Great find :)

carolanitz · 2018-04-02T10:08:45Z

Sign in to view

@@ -23,6 +23,7 @@ def iOS_pods
  pod 'MediaLibraryKit-prod'
  pod 'MobileVLCKit', '3.0.2'
  pod 'GTMAppAuth'
+  pod 'SimulatorStatusMagic', :configurations => ['Debug']


correct! @dcordero correct me if I'm wrong here please.
I think instead of importing it directly you have an extension on Appdelegate for the tests and that extension is only part of the testmodule. You would also not need the debug configuration that way. Though I'm not sure how that works with classmethods. I would take a look at Artsys App here

carolanitz · 2018-04-02T10:08:45Z

Sign in to view

+
+class VLCiOSTestMenu: XCTestCase {
+    let app = XCUIApplication()
+    let moreTab = XCUIApplication().tabBars.buttons.element(boundBy: 4)


Is it possible to get the tab by localized title instead ?

There is no localizable string for this :(
I think the "More" tab is using Apple's built-in localized strings.

carolanitz · 2018-04-02T10:08:45Z

Sign in to view

+    }
+
+    func localizedString(key: String) -> String {
+        let localizationBundle = Bundle(path: Bundle(for: VLCiOSTestMenu.self).path(forResource: deviceLanguage, ofType: ".lproj")!)!


force unwrap is not great. How do you feel about making the tests fail if the localization file can't be loaded?

Additionally it should be enough to load the bundle once for every language. That would speed up the test

carolanitz · 2018-04-02T10:08:45Z

Sign in to view

+    }
+
+    func testNavigationToVideoTab() {
+        app.tabBars.buttons["Video"].tap()


Is there no localizable string ?

carolanitz · 2018-04-02T10:08:45Z

Sign in to view

+    func localizedString(key: String) -> String {
+        let localizationBundle = Bundle(path: Bundle(for: VLCiOSTestMenu.self).path(forResource: deviceLanguage, ofType: ".lproj")!)!
+        let result = NSLocalizedString(key, bundle: localizationBundle, comment: "")
+        return result


seeing that this is used here as well it might be even good to have a testhelpers method ? We should definitely not duplicate the code here

carolanitz · 2018-04-02T10:08:45Z

Sign in to view

@@ -0,0 +1,13 @@
+ devices([
+   "iPhone X",
+   "iPhone 8"


I think we will need all the devices :D

carolanitz · 2018-04-02T10:08:45Z

Sign in to view

+ languages([
+   "en",
+   "de",
+   "fr",


we will also need all the languages that we support. And yes I know this will take forever

To create the framed screenshots, we need localized keyword.string and title.string for each language. Maybe this should be a progressive effort once we officially decide on what text to put on the framed screenshots??

absolutely let's just ask Louis!

carolanitz · 2018-04-02T10:08:45Z

fastlane/SnapshotHelper.swift

+    }
+}
+
+// Please don't remove the lines below


I'm expecting that you double checked and understood this file btw :)

carolanitz

we should switch to the accessibility element identifiers. that will resolve the bundle and localization issues. Also the simulator magic only as part of the Testtarget and adding it in the setup for the tests should resolve the import conundrum

carolanitz · 2018-04-03T14:01:04Z

Sign in to view

+        XCUIDevice.shared.orientation = .portrait
+        setupSnapshot(app)
+        helper = TestHelper(lang: deviceLanguage, target: VLCiOSTestMenu.self)
+        app.launch()


This here would actually be a better place for the status magic (but remember to disable it again in tearDown()).

carolanitz · 2018-04-03T14:01:04Z

Sign in to view

+import XCTest
+
+struct TestHelper {
+    let localizationBundle: Bundle


something that I didn't think about in my earlier review but that a friend pointed out is to use accessibility element identifiers. They don’t need to be localized! Since we need to make the app accessibility friendly anyway this will kill two birds with one stone :)

So it looks like none of the elements in the app has its accessibilityIdentifier set. How should we go about this? Update every single element's accessibilityIdentifier?

yes exactly. It needs to be added

carolanitz · 2018-04-03T14:01:04Z

Sign in to view

+        helper.tap(tabDescription: audio, app: app)
+        XCTAssertNotNil(app.navigationBars[audio])
+
+        snapshot("audio_tab")


btw could we separate the Unit Tests from the snapshot taking?

carolanitz · 2018-04-04T21:41:03Z

Sign in to view

 fastlane/test_output

+fastlane/screenshots/*/*.png
+fastlane/screenshots/screenshots.html


will you hate me if I ask for an alphabetically ordered .gitignore 🙊? At least within the fastlane section

Not at all 😄

carolanitz · 2018-04-09T12:12:14Z

looks already better. Could you use the accessibilityidentifier on the tabbaritems and other views as well? We need to avoid all use of the localization bundle otherwise renaming a tab would lead to failing tests for no reason

carolanitz · 2018-04-09T12:54:08Z

There is also something misconfigured in the xcodeproj for me. Might be just me though

carolanitz · 2018-04-09T13:02:53Z

fastlane/Snapfile

+   "iPad Pro (12.9-inch)", 
+   "iPad Pro (12.9-inch) (2nd generation)", 
+   "iPad Pro (10.5-inch)"
+ ])


carolanitz · 2018-04-09T13:17:54Z

the tests fail on iPad btw but the more tab will disappear anyway soon. And one more question. What are the screenshots doing for a language that is not defined? can we make sure that it uses the screenshots from the English version?

mkchoi212 · 2018-04-09T13:47:12Z

@carolanitz Right now, frameitdoes not work with any language that does not have their respective .string files. But let me see if I can make the default language English for them!

mkchoi212 · 2018-04-12T01:41:00Z

Please note that because the "More" tab is going to be removed soon, the test does not take the "More" tab into consideration. Because of this, the tests fail for all iPhones but succeed for all iPads

carolanitz · 2018-04-13T12:52:10Z

the merge conflicts just need to be resolved and I think this is good to go

mkchoi212 · 2018-04-16T20:53:11Z

Fixed all the merge conflicts!
Sorry for such a huge PR but thanks for the comments @dcordero @carolanitz ❤️
I'll make sure to break up these bad boys into smaller chunks next time!

Perfect status bar displays 100% battery life and 9:41 as the time. This allows for consistent, and perfect screenshoting, 100% of the time

To take framed screenshots, run `fastlane screenshots` Note that the list of supported devices and languages is random for now.

Notice that screenshots for fastlane are taken in the test file by calling `snapshot("IMG_NAME")`

Note that only German, French, and English have been added so far. Design/content for the framed screenshots are BASIC and NEEDS IMPROVEMENT.

iOS test cases are now compatible with both iPhone and iPad.

Note that a new target has been created in the Podfile specifically for iOS UITests.

Fixed bugs regarding change of localization during screenshoting

Note that string literals are being used all over the project to create identifiers. A new file containing all identifier literals should be created in the future.

Fastlane does not support default langauges. Therefore, in order to generate screenshots for all languages, default English keyword.strings and title.strings have been copied for all langauges.

mkchoi212 · 2018-04-18T19:08:07Z

All commits have been squashed!

mkchoi212 · 2018-04-18T19:10:42Z

VLC.xcodeproj/project.pbxproj

@@ -3431,7 +3408,6 @@
 				ALWAYS_EMBED_SWIFT_STANDARD_LIBRARIES = YES;
 				CLANG_ANALYZER_NONNULL = YES;
 				CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
-				CLANG_CXX_LANGUAGE_STANDARD = "gnu++14";


@carolanitz, this fixes the misconfigured in the xcodeproj problem that you discovered!

dcordero reviewed Apr 1, 2018

View changes

mkchoi212 force-pushed the mkchoi212:master branch Apr 1, 2018

carolanitz requested changes Apr 2, 2018

View changes

mkchoi212 force-pushed the mkchoi212:master branch Apr 3, 2018

carolanitz requested changes Apr 3, 2018

View changes

mkchoi212 force-pushed the mkchoi212:master branch 2 times, most recently Apr 4, 2018

carolanitz reviewed Apr 4, 2018

View changes

mkchoi212 force-pushed the mkchoi212:master branch 3 times, most recently Apr 5, 2018

carolanitz reviewed Apr 9, 2018

View changes

fastlane/Snapfile

"iPad Pro (12.9-inch)",

"iPad Pro (12.9-inch) (2nd generation)",

"iPad Pro (10.5-inch)"

])

This comment has been minimized.

Sign in to view

carolanitz Apr 9, 2018
Member

👍

dcordero mentioned this pull request Apr 9, 2018

Add Ruby and Swift linters and PR template #27

Closed

5 of 5 tasks complete

mkchoi212 force-pushed the mkchoi212:master branch Apr 12, 2018

mkchoi212 force-pushed the mkchoi212:master branch 4 times, most recently Apr 12, 2018

carolanitz approved these changes Apr 13, 2018

View changes

mkchoi212 force-pushed the mkchoi212:master branch 2 times, most recently Apr 13, 2018

mkchoi212 added 9 commits Apr 1, 2018

[Simulator] Add perfect status bar

78f7053

Perfect status bar displays 100% battery life and 9:41 as the time. This allows for consistent, and perfect screenshoting, 100% of the time

[fastlane] Enable `fastlane screenshots`

975ca70

To take framed screenshots, run `fastlane screenshots` Note that the list of supported devices and languages is random for now.

[UI] New tests for the new UI

050b23f

Notice that screenshots for fastlane are taken in the test file by calling `snapshot("IMG_NAME")`

[fastlane] Add frameit keyword/title data

a19889e

Note that only German, French, and English have been added so far. Design/content for the framed screenshots are BASIC and NEEDS IMPROVEMENT.

[UI] Modify iOS test cases for iPad

731460b

iOS test cases are now compatible with both iPhone and iPad.

[Simulator] Relocate perfect status bar

7e6593d

Note that a new target has been created in the Podfile specifically for iOS UITests.

[fastlane] Seperate screenshot code from testing code

773b500

Fixed bugs regarding change of localization during screenshoting

[UI] Use accessibilityIdentifier to interact

227814f

Note that string literals are being used all over the project to create identifiers. A new file containing all identifier literals should be created in the future.

[UI] Add default keyword and title files for all languages.

Loading status checks…

e62a7fb

Fastlane does not support default langauges. Therefore, in order to generate screenshots for all languages, default English keyword.strings and title.strings have been copied for all langauges.

mkchoi212 force-pushed the mkchoi212:master branch to e62a7fb Apr 18, 2018

mkchoi212 reviewed Apr 18, 2018

View changes

vlc-mirrorer merged commit e62a7fb into videolan:master Apr 18, 2018
1 check failed

1 check failed

ci/circleci Your tests failed on CircleCI
Details

videolan / vlc-ios

Automated framed screenshots #28

Automated framed screenshots #28

Conversation

mkchoi212 commented Apr 1, 2018 • edited

mkchoi212 commented Apr 1, 2018

This comment has been minimized.

dcordero Apr 1, 2018 Collaborator

This comment has been minimized.

dcordero Apr 1, 2018 Collaborator

This comment has been minimized.

dcordero Apr 1, 2018 Collaborator

This comment has been minimized.

dcordero Apr 1, 2018 Collaborator

This comment has been minimized.

dcordero Apr 1, 2018 Collaborator

This comment has been minimized.

dcordero Apr 1, 2018 Collaborator

This comment has been minimized.

mkchoi212 Apr 1, 2018 Author Collaborator

This comment has been minimized.

carolanitz Apr 2, 2018 • edited Member

dcordero commented Apr 1, 2018 • edited

dcordero commented Apr 2, 2018

carolanitz left a comment

This comment has been minimized.

carolanitz Apr 2, 2018 Member

This comment has been minimized.

mkchoi212 Apr 2, 2018 Author Collaborator

This comment has been minimized.

carolanitz Apr 3, 2018 Member

This comment has been minimized.

carolanitz Apr 2, 2018 • edited Member

This comment has been minimized.

carolanitz Apr 2, 2018 Member

This comment has been minimized.

mkchoi212 Apr 2, 2018 • edited Author Collaborator

This comment has been minimized.

carolanitz Apr 2, 2018 Member

This comment has been minimized.

carolanitz Apr 2, 2018 Member

This comment has been minimized.

carolanitz Apr 2, 2018 Member

This comment has been minimized.

carolanitz Apr 2, 2018 Member

This comment has been minimized.

carolanitz Apr 2, 2018 Member

This comment has been minimized.

carolanitz Apr 2, 2018 Member

This comment has been minimized.

mkchoi212 Apr 3, 2018 Author Collaborator

This comment has been minimized.

carolanitz Apr 3, 2018 Member

This comment has been minimized.

carolanitz Apr 2, 2018 Member

carolanitz left a comment

This comment has been minimized.

carolanitz Apr 3, 2018 Member

This comment has been minimized.

carolanitz Apr 3, 2018 Member

This comment has been minimized.

mkchoi212 Apr 4, 2018 Author Collaborator

This comment has been minimized.

carolanitz Apr 4, 2018 Member

This comment has been minimized.

carolanitz Apr 3, 2018 Member

This comment has been minimized.

carolanitz Apr 4, 2018 • edited Member

This comment has been minimized.

mkchoi212 Apr 5, 2018 Author Collaborator

carolanitz commented Apr 9, 2018

carolanitz commented Apr 9, 2018

This comment has been minimized.

carolanitz Apr 9, 2018 Member

carolanitz commented Apr 9, 2018

mkchoi212 commented Apr 9, 2018

mkchoi212 commented Apr 12, 2018

carolanitz commented Apr 13, 2018

mkchoi212 commented Apr 16, 2018

mkchoi212 commented Apr 18, 2018

mkchoi212 commented Apr 1, 2018 •

edited

dcordero Apr 1, 2018
Collaborator

dcordero Apr 1, 2018
Collaborator

dcordero Apr 1, 2018
Collaborator

dcordero Apr 1, 2018
Collaborator

dcordero Apr 1, 2018
Collaborator

dcordero Apr 1, 2018
Collaborator

mkchoi212 Apr 1, 2018
Author Collaborator

carolanitz Apr 2, 2018 •

edited

Member

dcordero commented Apr 1, 2018 •

edited

carolanitz Apr 2, 2018
Member

mkchoi212 Apr 2, 2018
Author Collaborator

carolanitz Apr 3, 2018
Member

carolanitz Apr 2, 2018 •

edited

Member

carolanitz Apr 2, 2018
Member

mkchoi212 Apr 2, 2018 •

edited

Author Collaborator

carolanitz Apr 2, 2018
Member

carolanitz Apr 2, 2018
Member

carolanitz Apr 2, 2018
Member

carolanitz Apr 2, 2018
Member

carolanitz Apr 2, 2018
Member

carolanitz Apr 2, 2018
Member

mkchoi212 Apr 3, 2018
Author Collaborator

carolanitz Apr 3, 2018
Member

carolanitz Apr 2, 2018
Member

carolanitz Apr 3, 2018
Member

carolanitz Apr 3, 2018
Member

mkchoi212 Apr 4, 2018
Author Collaborator

carolanitz Apr 4, 2018
Member

carolanitz Apr 3, 2018
Member

carolanitz Apr 4, 2018 •

edited

Member

mkchoi212 Apr 5, 2018
Author Collaborator

carolanitz Apr 9, 2018
Member

mkchoi212 Apr 18, 2018
Author Collaborator