Update template to include test for releaseable builds #92

srirambv · 2018-09-10T21:55:07Z

No description provided.


        Update template to include test for releaseable builds

bbondy · 2018-09-11T01:47:21Z

bc-wikitemplate.md

+
+### Crash verification
+- [ ] Verify these [crash issues](https://github.com/brave/brave-browser/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A%22Releasable+builds+0.55.x%22+label%3Acrash+is%3Aclosed++label%3AQA%2FNeeded) have not regressed 
+- [ ] Verify these [webview crash issues](https://github.com/brave/brave-browser/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A%22Releasable+builds+0.55.x%22+label%3Acrash/webview+is%3Aclosed++label%3AQA%2FNeeded) have not regressed 


I don't think we're at risk in general of the crashes regressing, so I don't think we need to spend time on these on each release.

Agreed, not sure if it's the best idea keep going through all those crashes to make sure they haven't regressed. Getting stats.brave into a better state would be a better idea to keep an eye on crash rates coming in on the different channels.

bbondy · 2018-09-11T01:48:20Z

bc-wikitemplate.md

 - [ ] Test that an old extension gets updated to a new one from an old profile on Google server
 - [ ] Test that an old extension gets updated to a new one from an old profile on Brave server
+- [ ] Verify magnet links loads Torrent viewer page and able to download torrent


You can add .torrent files too, that will land probably today.

bbondy · 2018-09-11T01:48:58Z

bc-wikitemplate.md

- [ ] Test that you can log into Netflix and start a show from a fresh profile
+### CWS
+
+- [ ] Verify installing uBO/ABP from CWS shows warning message `NOT A RECOMMENDED BRAVE EXTESION!`but still gets installed


I think you can just pick a single extension. Doesn't have to be uBlock or ABP either.

bbondy · 2018-09-11T01:50:43Z

bc-wikitemplate.md

+
+- [ ] Verify installing uBO/ABP from CWS shows warning message `NOT A RECOMMENDED BRAVE EXTESION!`but still gets installed
+- [ ] Verify installing LastPass from CWS doesn't show any warning message 
+- [ ] Verify installing an extesion that is not vetted by Brave gets blocked 


not vetted extensions can still be installed, just gives a warning. We don't have any blocked yet so probably just remove this?

I think we should keep this but change the case to make sure that none vetted extensions correctly warn the user before installation. Risk missing a regression where warnings stop working. Could be really bad if users start complaining that Brave didn't warn them and they installed a bad actor extension.

Would suggest to test all three allowed/vetted but allowed/ blocked just so we cover all possible scenarios

bbondy · 2018-09-11T01:51:05Z

bc-wikitemplate.md

+### PDF
+
+- [ ] Test that you can print a PDF
+- [ ] Test that PDF is loaded over https at https://basicattentiontoken.org/BasicAttentionTokenWhitePaper-4.pdf


bbondy · 2018-09-11T01:52:24Z

bc-wikitemplate.md

+
+#### Adblock
+
+- [ ] Verify the execeptions in `brave/common/shield_exceptions.cc` work 


We have automated for this so you can probably just remove this.

bbondy · 2018-09-11T01:53:04Z

bc-wikitemplate.md

+
+## Yet to implement
+
+- Keyboard Shortcuts


What's left for that?

Custom shortcuts that would be added for Brave. Tor tab, preferences, reopen last window etc...

bbondy

Some inline comments

kjozwiak · 2018-09-11T03:50:27Z

bc-wikitemplate.md

+- [ ] Verify these [webview crash issues](https://github.com/brave/brave-browser/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A%22Releasable+builds+0.55.x%22+label%3Acrash/webview+is%3Aclosed++label%3AQA%2FNeeded) have not regressed 
+
+
+### Installer

 - [ ] Check that installer is close to the size of last release.


@bbondy will we be mostly using stub installers? If that's the case, we'll need to change this as it's not really a valid check as the application we'll be pulled/installed from a remote server. If we're going to support both regular binaries and stub installers, we should add another case similar to the following:

Install brave using the stub installer and ensure the installed profile is similar in size to the previous version

I retained that test for the offline installer for Windows. These setup related tests are blocked on brave/omaha#4

kjozwiak · 2018-09-11T04:13:39Z

bc-wikitemplate.md

+- [ ] Verify these [webview crash issues](https://github.com/brave/brave-browser/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A%22Releasable+builds+0.55.x%22+label%3Acrash/webview+is%3Aclosed++label%3AQA%2FNeeded) have not regressed 
+
+
+### Installer

 - [ ] Check that installer is close to the size of last release.
 - [ ] Check signature: If OS Run `spctl --assess --verbose /Applications/Brave-Browser.app/` and make sure it returns `accepted`.  If Windows right click on the installer exe and go to Properties, go to the Digital Signatures tab and double click on the signature.  Make sure it says "The digital signature is OK" in the popup window.


I know the current QA team will know that the spctl --assess --verbose /Applications/Brave-Browser.app/ might differ while on different channels but we should explicitly mention which locations need to be checked depending on the channel. Example: spctl --assess --verbose /Applications/Brave-Browser-Dev.app/. I think it will also help contributors once we start trying to get more folks involved in brave-core.

@bbondy @srirambv thoughts?

We should also find out a way if there's a way of checking the signatures on Linux (at least Ubunutu).

Agreed. changed it to beta since that is where most of the feature/regression fix testing will happen once we have proper channels for testing.

kjozwiak · 2018-09-11T04:15:56Z

bc-wikitemplate.md

+- [ ] Verify these [webview crash issues](https://github.com/brave/brave-browser/issues?utf8=%E2%9C%93&q=is%3Aissue+milestone%3A%22Releasable+builds+0.55.x%22+label%3Acrash/webview+is%3Aclosed++label%3AQA%2FNeeded) have not regressed 
+
+
+### Installer

 - [ ] Check that installer is close to the size of last release.
 - [ ] Check signature: If OS Run `spctl --assess --verbose /Applications/Brave-Browser.app/` and make sure it returns `accepted`.  If Windows right click on the installer exe and go to Properties, go to the Digital Signatures tab and double click on the signature.  Make sure it says "The digital signature is OK" in the popup window.
 - [ ] Check Brave and Chromium version in chrome://about


Should be chrome://version/ and not chrome://about

Removed it as it was checked again in about page section

kjozwiak · 2018-09-11T04:19:26Z

bc-wikitemplate.md

- [ ] Check that you can import from Google Chrome (We implemented all ourselves)
- [ ] Check that you can import from Firefox (Cookies is all we implemented ourselves)
- [ ] Check that you can import from Edge (No custom code we implemented ourselves)
+- [ ] Test chrome://adblock loads adblock page


nit: easier to read while going through the manual passes if formatted chrome://adblock

kjozwiak · 2018-09-11T04:19:36Z

bc-wikitemplate.md

- [ ] Check that you can import from Firefox (Cookies is all we implemented ourselves)
- [ ] Check that you can import from Edge (No custom code we implemented ourselves)
+- [ ] Test chrome://adblock loads adblock page
+- [ ] Test chrome://rewards loads Brave rewards page 


nit: easier to read while going through the manual passes if formatted chrome://rewards

kjozwiak · 2018-09-11T04:19:44Z

bc-wikitemplate.md

- [ ] Check that you can import from Edge (No custom code we implemented ourselves)
+- [ ] Test chrome://adblock loads adblock page
+- [ ] Test chrome://rewards loads Brave rewards page 
+- [ ] Test chrome://newtab loads a new tab


nit: easier to read while going through the manual passes if formatted chrome://newtab

kjozwiak · 2018-09-11T04:19:54Z

bc-wikitemplate.md

+- [ ] Test chrome://adblock loads adblock page
+- [ ] Test chrome://rewards loads Brave rewards page 
+- [ ] Test chrome://newtab loads a new tab
+- [ ] Test chrome://welcome loads the welcome page 


nit: easier to read while going through the manual passes if formatted chrome://welcome

kjozwiak · 2018-09-11T04:20:04Z

bc-wikitemplate.md

+- [ ] Test chrome://rewards loads Brave rewards page 
+- [ ] Test chrome://newtab loads a new tab
+- [ ] Test chrome://welcome loads the welcome page 
+- [ ] Test chrome://version correctly shows Brave version and Chromium version 


nit: easier to read while going through the manual passes if formatted chrome://version

kjozwiak · 2018-09-11T04:27:19Z

bc-wikitemplate.md

+
+- [ ] Test that you can print a PDF
+- [ ] Test that PDF is loaded over https at https://basicattentiontoken.org/BasicAttentionTokenWhitePaper-4.pdf
+- [ ] Test that PDF is loaded over http at http://www.pdf995.com/samples/pdf.pdf


same nit as @bbondy above: HTTP (protocols should always be capitalized)


        Update based on feedback and fixed test run generator

srirambv · 2018-09-11T14:25:49Z

ready for review

bbondy · 2018-09-11T16:26:59Z

bc-wikitemplate.md

+- Sync
+- Tor private tabs
+- Manual Update tests from older version
+  - Test that updating using `BRAVE_UPDATE_VERSION=0.8.3` env variable works correctly.


Is this valid in this codebase? I don't think so.

No just kept it for reference to verify for offline installer once that is available

Update template to include test for releaseable builds

f97cc00

srirambv added the testsheet label Sep 10, 2018

srirambv self-assigned this Sep 10, 2018

srirambv requested review from bbondy and kjozwiak Sep 10, 2018

bbondy reviewed Sep 11, 2018

View changes

bc-wikitemplate.md Outdated

### PDF

- [ ] Test that you can print a PDF

- [ ] Test that PDF is loaded over https at https://basicattentiontoken.org/BasicAttentionTokenWhitePaper-4.pdf

This comment has been minimized.

Sign in to view

Copy link

bbondy Sep 11, 2018
Member

nit: HTTPS

bbondy reviewed Sep 11, 2018

View changes

bc-wikitemplate.md Outdated

#### Adblock

- [ ] Verify the execeptions in `brave/common/shield_exceptions.cc` work

This comment has been minimized.

Sign in to view

Copy link

bbondy Sep 11, 2018
Member

We have automated for this so you can probably just remove this.

bbondy reviewed Sep 11, 2018

View changes

bbondy requested changes Sep 11, 2018

View changes

kjozwiak reviewed Sep 11, 2018

View changes

Update based on feedback and fixed test run generator

616ef35

bbondy reviewed Sep 11, 2018

View changes

bbondy approved these changes Sep 11, 2018

View changes

srirambv merged commit a1574b3 into brave:master Sep 12, 2018

This was referenced Sep 12, 2018

Revert "Update template to include test for releaseable builds" #97

Merged

Update template for b-c tests #98

Merged

brave / qa-resources

Update template to include test for releaseable builds #92

Update template to include test for releaseable builds #92

srirambv commented Sep 10, 2018

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bbondy left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

srirambv commented Sep 11, 2018

This comment has been minimized.

This comment has been minimized.


		#### Adblock

		- [ ] Verify the execeptions in `brave/common/shield_exceptions.cc` work

brave / qa-resources

Join GitHub today

Update template to include test for releaseable builds #92

Update template to include test for releaseable builds #92

Conversation

srirambv commented Sep 10, 2018

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bbondy left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

kjozwiak Sep 11, 2018 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

srirambv commented Sep 11, 2018

This comment has been minimized.

This comment has been minimized.

kjozwiak Sep 11, 2018 •

edited