Add metric to help assess awareness of init script customizability

[jasonrudolph]

Description of the Change

Building on the updates in #90 and #94, this pull request continues our quest to better understand the awareness and approachability of Atom's "hackability."

Atom's hackability comes in multiple forms, and the init script (i.e., ~/.atom/init.coffee or ~/.atom/init.js) is arguably the most powerful form of customizability, and it's a great sandbox for trying out customizations that could eventually graduate into a full-fledged package. The Flight Manual describes the steps for customizing the init script, but we don't currently have a way of knowing whether that translates into users actually performing any customization. To assess the awareness and approachability of customizing the init script, this pull request adds a metric to report an event when the user changes their init script.

Equipped with this information, we'll be able to assess the effectiveness of our efforts to increase the awareness and approachability of init script customization. For example, does updating the welcome guide's description of the init script increase or decrease the percentage of users that customize their init script?

Design

Implementation-wise, this pull request watches for the user's init script to get opened in a TextEditor in Atom. Any time that TextEditor is saved, we record an event indicating that the user has changed their init script.

Alternate Designs

• We could use pathwatcher to watch for changes to the init script (ddcb6cd). With that approach, as long as Atom is open, we would record an event any time the init script changes on disk, regardless of whether the user changed it in Atom or in another editor. However, if the user has multiple Atom windows open, we would record an event for each open Atom window. 😬
• At startup, we could compare the contents of the user's init script to the default init script. If the user's init script is different than the default init script, then we could report that the user has customized their init script. However, if we ever change the default init.coffee, this approach would wrongly conclude that an existing user has customized their init script when they have an unedited local copy of the old default init script. 🙈
• We could attempt to detect the existence of code in the init script as opposed to just detecting arbitrary changes to the init script. For example, we could try to detect new commands that get registered, new observers that get registered, etc. However, given that you can do almost anything in the init script, a user can make meaningful customizations that don't register commands, observers, etc. Therefore, any specific set of things we detect (e.g., commands, observers) will be an incomplete set of the possible customizations, so we won't have an accurate measure of which users have customized their init script. 😦
• We could attempt to detect usage of commands that were added in init script. Instead of simply detecting that a user customized their init script at some point in the past, this would be one way to measure that a user is using those customizations. However, as noted in the previous bullet, commands are just one form of customization. If we only measure usage of custom commands, we wouldn't have an accurate measure of the number of users that are customizing their init script. 😕

Possible Drawbacks

• Measuring that the user saved their init script doesn't mean that they're using the customizations they've made to their init script. For example, the user could add a command in their init script and then never use that command.
• The user could use a different editor to customize their Atom init script. If so, this implementation won't report that the user has customized their init script.
• If a user makes only trivial changes (e.g., whitespace changes) to their init script and then saves the file, this implementation will report that that the user has customized their init script, even though their edits haven't actually customized Atom's behavior.
• The current implementation records an event each time the init script is saved. If the user saves the file multiple times while making a single conceptual change, Atom will record multiple events (i.e., one event each time the script is saved).

Verification Process

• Verify that both init.coffee and init.js are supported
  • Edit and save your init.coffee file, and verify that a userInitScriptChanged event is recorded
  • Edit and save your init.js file, and verify that a userInitScriptChanged event is recorded
• Verify that Windows paths and macOS/Linux paths are supported
  • Edit and save your init script on macOS, and verify that a userInitScriptChanged event is recorded
  • Edit and save your init script on Windows, and verify that a userInitScriptChanged event is recorded
• Launch Atom with a custom home directory (e.g., ATOM_HOME=/tmp/some-nonstandard-dir), edit and save your init script, and verify that a userInitScriptChanged event is recorded

TODO

• Record event when init script changes
• Document and execute verification process

[jasonrudolph]

@daviwil @maxbrunsfeld: What do y'all think of this approach to measuring awareness/approachability of customizing the init script? This approach has some drawbacks as noted in the pull request body, and I'm wondering if there's a different approach that you can think of that might be a better fit.

[daviwil]

Yeah, it's going to be difficult to get a highly accurate idea of whether they're actually using using their customizations. I think it's enough to know that they have an init.js file in their Atom folder because it shows they at least went that far. I also like the idea of logging when the user saves a change to their init file in Atom.

Would it make sense to have two data points, one for merely having an init.js file and one for interacting with it in the editor? I'm thinking of the case of a user who has an existing init.js file stored in a .dotfiles repo that they've customized a while back and don't actively make new edits to it.

[daviwil]

Code looks great! So happy that we're getting some new and meaningful metrics added now 🙏

[annthurium]

Looks great, @jasonrudolph! Thanks for tackling this.

If we want to know if users are using custom commands, we could track that as well. We are already sending an event for each command. If we can generate a list of known, non-custom commands, we can filter the custom ones from that list on the back end. Something to discuss with telliott27, anyway.

[annthurium]

nit: should temp be a dev dependency? It looks like it's only used in tests.

[maxbrunsfeld]

This looks great!

As a possible future enhancement, we could report the number of commands that are defined every time an init script is loaded. I think this would require adding three new event APIs to core: atom.onWillLoadUserInitScript and atom.onDidLoadUserInitScript (emitted here), and atom.commands.onDidAddCommand (emitted here).

If we did that, I think it'd even be possible to report a metric every time the user executes a command that was defined in their init script.

[jasonrudolph]

Would it make sense to have two data points, one for merely having an init.js file and one for interacting with it in the editor? I'm thinking of the case of a user who has an existing init.js file stored in a .dotfiles repo that they've customized a while back and don't actively make new edits to it.

@daviwil: I think we'd see that all users have an init script present on disk. The first time you run Atom, it creates your Atom home directory and populates it with a handful of default files, including init.coffee:

$ ls ~/.atom
blob-store	github.cson	packages	styles.less
compile-cache	init.coffee	snippets.cson
config.cson	keymap.cson	storage

$ cat .atom/init.coffee
# Your init script
#
# Atom will evaluate this file each time a new window is opened. It is run
# after packages are loaded/activated and after the previous editor state
# has been restored.
#
# An example hack to log to the console when each text editor is saved.
#
# atom.workspace.observeTextEditors (editor) ->
#   editor.onDidSave ->
#     console.log "Saved! #{editor.getPath()}"

If we want to know if users are using custom commands, we could track that as well. -- @annthurium

As a possible future enhancement, we could report the number of commands that are defined every time an init script is loaded ... If we did that, I think it'd even be possible to report a metric every time the user executes a command that was defined in their init script. -- @maxbrunsfeld

@annthurium @maxbrunsfeld: I dig it! Those things are definitely on my wish list as we look ahead to future enhancements. 🤞

[daviwil]

🤦‍♂️ I ls'ed my ~/.atom folder and my eyes were expecting to see init.js so I think I skimmed over it. Thanks for pointing that out!

[jasonrudolph]

• Document and execute verification process

I updated the pull request body to describe the verification process.