Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metric to help assess awareness of package ecosystem #90

Merged
merged 3 commits into from Jun 20, 2018

Conversation

Projects
None yet
4 participants
@jasonrudolph
Copy link
Member

jasonrudolph commented Jun 19, 2018

Description of the Change

We consider Atom's extensibility (or "hackability!" 馃榿) to be one of its most important features. That extensibility comes in multiple forms, but for many users, the package ecosystem will be the primary means of extensibility. Atom's welcome guide promotes the package ecosystem, but we don't currently have a way of knowing whether that translates into users actually installing packages. To assess the awareness and approachability of the package ecosystem, this pull request adds a metric to report the number of packages that a user has installed.

Equipped with this information, we'll be able to assess the effectiveness of our efforts to increase the awareness and approachability of the package ecosystem. For example, does a particular tweak to the welcome guide increase or decrease the percentage of users that have installed a package? Similarly, if we were to add an "Install" button on each package page on https://atom.io, does that result in an increase in the percentage of users that have installed a package?

Implementation-wise, this pull request sends an event that reports the quantity of optional (i.e., non-bundled) packages that get activated at Atom startup.

Alternate Designs

Instead of sending the count of optional packages that are activated, it would provide richer information if we sent the names of the optional packages that are activated. However, I don't think it will be practical to take that approach:

  • Our analytics system doesn't currently support metrics that specific a list (or array) of data, so we can't report a single metric that includes the list of packages. (I suppose we could send a string of package names separated by some delimiter, but ... yuck.)
  • To get around that limitation, we could report an event for each package that gets activated and we could include the name of the activated package. However, that would likely result in many more network requests (since our analytics library makes an HTTP request for each event being reported), and we'd likely exceed the rate limit, causing some metrics to get dropped.

Verification Process

  • Using an Atom installation that only has Atom's bundled packages, verify that an event is recorded at startup reflecting that the instance has zero optional packages
  • Using an Atom installation that has optional packages installed, verify that an event is recorded at startup reflecting the number of optional packages
@jasonrudolph

This comment has been minimized.

Copy link
Member Author

jasonrudolph commented Jun 19, 2018

Looks like CI doesn't like the new tests. I'll look into those failures and see what I can figure out.

521845_1

@annthurium
Copy link
Contributor

annthurium left a comment

looks good to me (once tests are passing)


Reporter.sendEvent(
'package',
'numberOptionalPackagesActivatedAtStartup',

This comment has been minimized.

@annthurium

annthurium Jun 19, 2018

Contributor

yeah, it's too bad it would be tricky to send the package names here, as that would be interesting.

This comment has been minimized.

@damieng

damieng Jun 19, 2018

Contributor

I think the other problem with sending package names is that for new packages it's effectively personally identifying information - the package author.


// Mimic the event that is emitted when Atom finishes loading all
// packages at startup. (We don't want to weigh down this test with the
// overhead of actually load _all_ packages.)

This comment has been minimized.

@annthurium

annthurium Jun 19, 2018

Contributor

thanks for the comment - that's really helpful!

@maxbrunsfeld
Copy link
Contributor

maxbrunsfeld left a comment

馃憤 Seems like a good additional data point.

@@ -0,0 +1,3 @@
module.exports = {
activate () {}
}

This comment has been minimized.

@maxbrunsfeld

maxbrunsfeld Jun 19, 2018

Contributor

I think that in order to activate, this package might need a package.json. Alternatively, you could probably use one of the existing packages in the fixtures/packages folder for this purpose.

This comment has been minimized.

@jasonrudolph

jasonrudolph Jun 20, 2018

Author Member

Thanks for the suggestion, @maxbrunsfeld. It seems to have been an issue with the package directory path. a802ba3 resolves the issue. I'm not sure why I didn't need to set that path for the tests to pass on my local dev machine, but at least now the tests pass locally and on CI. 馃槄

Fix Travis CI failure? 馃檹馃
Attempt to resolve failure seen at 
https://travis-ci.org/atom/metrics/builds/394289538#L397.

Before activating the fixture package, add its parent directory to the 
package directory paths.
@jasonrudolph

This comment has been minimized.

Copy link
Member Author

jasonrudolph commented Jun 20, 2018

I updated the pull request body to describe the verification process, and the screenshot below shows the metrics resulting from my testing.

You can view metrics by various dimensions, and this particular view shows the average number of optional packages activated by Atom version. In a VM running an Atom 1.29 dev build, you can see that I have no optional packages installed. And on my main dev machine running an Atom 1.30 dev build, you can see that I have 38 optional packages installed.

analytics_360

@jasonrudolph jasonrudolph merged commit 71b77c7 into master Jun 20, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@jasonrudolph jasonrudolph deleted the understand-package-ecosystem-awareness branch Jun 20, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can鈥檛 perform that action at this time.