Skip to content
This repository was archived by the owner on Jun 21, 2023. It is now read-only.

Conversation

jcansdale
Copy link
Collaborator

@jcansdale jcansdale commented Sep 21, 2018

What this PR does

  • Show clone URL tab by default 50% of the time
  • Increment NumberOfCloneView*Tab counter when Clone dialog is displayed
  • Record GitHub and GitHub Enterprise clones separately
  • Add the following counters
NumberOfCloneViewGitHubTab
NumberOfCloneViewEnterpriseTab
NumberOfCloneViewUrlTab
NumberOfGitHubClones
NumberOfEnterpriseClones

TODO

  • Add separate counters for NumberOfGitHubClones and NumberOfEnterpriseClones

A/B Testing

The counters are intended to answer the following questions about showing the clone URL tab by default:

  1. Does it make users more less likely to clone a repository
  2. Does it make users more less likely to use the clone dialog
  3. Is there any difference in behavior between GitHub and GitHub Enterprise

How to test

The Start Page button

  1. Open the file at %USERPROFILE%\AppData\Local\GìtHūbVisualStudio\user.json
  2. Users with a UserGuid that ends with an even number will be group A (and odd ones group B)
  3. Click on the Start Page > Open > GitHub button
  4. If you're in group A the URL tab should be selected
  5. If you're in group B the GitHub or Enterprise tab should be selected
    image
  6. Change the last number of your UserGuid, restart Visual Studio and repeat the steps above

The Team Explorer - Connect buttons

  1. Click the Manage Connections button on the Team Explorer tool bar
  2. Click Clone on either the GitHub or GitHub Enterprise section
  3. When the Clone dialog appears, the NumberOfCloneViewGitHubTab, NumberOfCloneViewEnterpriseTab or NumberOfCloneViewUrlTab counter should increment (depending on which tab is selected by default)
  4. Select and clone a repository
  5. The NumberOfClones and NumberOfGitHubClones / NumberOfEnterpriseClones counters should increment (depending on what type of repository as cloned)

image

Note: the counters will appear in the log when Enable Trace Logging is checked
image

Assumptions

  • The last number of the UserGuid is random

Related

Add the following counters for tracking the clone view being opened
with different selected tabs:

NumberOfCloneViewGitHubTab
NumberOfCloneViewEnterpriseTab
NumberOfCloneViewUrlTab
Check the correct default tab is being selected and the
NumberOfCloneView*Tab counters are being incremented.
@meaghanlewis meaghanlewis added this to the 2.5.6 milestone Sep 21, 2018
@jcansdale jcansdale force-pushed the fixes/327-default-url-tab-AB-testing branch from b72a552 to 8fa09e5 Compare September 22, 2018 08:26
Check NumberOfGitHubClones and NumberOfEnterpriseClones are being
incremented separately.
Expect failing tests on this commit.
Record GitHub and GitHub Enterprise clones separately.
When the assert fails it will look like this:
```
Test
'IncrementCounter_Showing_Default_Tab("https://enterprise.com",True,1,"xxx")'
 failed:
 NSubstitute.Exceptions.ReceivedCallsException : Expected to receive
exactly 1 call matching:
 IncrementCounter(x => (Convert(x.Body).Member.Name ==
value(GitHub.App.UnitTests.ViewModels.Dialog.Clone.RepositoryCloneViewModelTests+<>c__DisplayClass2_0).counterName))
Actually received no matching calls.
Received 1 non-matching call (non-matching arguments indicated with '*'
characters):
 IncrementCounter(*model => model.NumberOfCloneViewUrlTab*)
```
@meaghanlewis
Copy link
Contributor

This looks great to me, thanks @jcansdale! Really love all the new metrics about which tabs of the dialog are viewed/opened and the number of clones.

@jcansdale jcansdale merged commit c748a60 into master Sep 25, 2018
@jcansdale jcansdale deleted the fixes/327-default-url-tab-AB-testing branch September 25, 2018 10:16
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants