Typewriter 2.0 (npm: 7.0.0)

[colinking]

This PR is a complete rebuild of typewriter (called 2.0 or v7) that focuses on improving:

1. Developer experience (DX)
2. Reliability
3. Ease of contribution

Developer Experience

Typewriter is now built around the idea of a configuration file (typewriter.yml) that is used by the CLI to understand how to build your clients. This means that users no longer need to set 4-5 different flags (Tracking Plan ID, directory, workspace slug, etc.) each time they invoke the CLI (without which, the CLI tended to surface difficult-to-understand errors).

To get started, all a user has to know is a single command:

npx typewriter init

This will launch a wizard that walks a user through creating a typewriter.yml and then builds their first client.

From then on, a user can just run npx typewriter to pull down their latest Tracking Plan and rebuild their clients.

Through this config file, we are able to provide saner defaults, which in Segment's case, allowed us to remove ~30-50 lines of Makefile from every typewriter-enabled project.

The entire typewriter CLI was also rebuilt in Ink so we can provide a user-friendly CLI with a special focus on easy-to-remedy error handling (f.e., no stack traces).

We've also launched documentation on the Segment Docs site for typewriter: https://segment.com/docs/protocols/typewriter

Reliability

Previously, when we pushed out new typewriter builds, we would use a combination of snapshot testing on the generated clients and manual testing via a series of example apps in the typewriter repo. We could run the app, which would send off an event or two to Segment, and then we could verify that event showed up correctly in the Segment Debugger.

However, this was problematic because a) it wasn't automated, and therefore didn't always happen and b) it only tested 1-2 example events, so it wasn't exhaustive by any means.

Instead, we now ship a full suite of end-to-end tests that use segmentio/mock to capture analytics events fired by an example app, and then a standardized test suite (suite.test.ts) that inspects all of the captured events to verify they match a set of Joi schemas.

This suite, besides verifying that typewriter generates clients that successfully build, also tests for scenarios that have previously caused issues with new releases of typewriter such as:

• What happens if two events have event names that collide?
• What happens if an event has no explicit properties?
• Is null sent to Segment for nullable properties?
• ... etc.

A large benefit of this setup is that the suite itself is language-agnostic, so a new end-to-end test suite can just be an arbitrary app with the Tracking API address updated to localhost:8765 (segmentio/mock). It otherwise requires no testing logic within the example app itself. 🎉

We've built out an e2e test suite for all supported languages, including analytics.js (in JS and TS), analytics-node (also in JS + TS) and analytics-ios (in Objective-C and Swift).

Ease of contribution

Adding support for a new language or SDK is now much more straightforward. We've overhauled how client generation works by moving off of QuickType (a powerful library for generating JSON serialization/deserialization clients). QuickType was a great starting point for us, but we had to extend their generators in fairly non-obvious ways to customize what code was generated (subclassing fields/logic that was split across the typewriter and QuickType repos, f.e.). This made our generators long, such as the iOS generator which was previously 600 lines (now <300!), and hard to follow -- especially for new contributors.

We decided to take a different approach to generation in favor of readability and reducing the amount of work+context each generator required. With this, we returned to generators based on Handlebars templates. Building a new generator simply involves implementing the various methods of a Generator to produce a set of objects that will be passed into your templates.

As part of removing QuickType, we had to roll our own implementations of a few components, specifically:

• AST: Previously, we interacted with QuickType's AST which is generalized to support multiple types of schemas (GraphQL, arbitrary JSON, etc.). In our case, we only need to support JSON Schema, so we built our own JSON Schema parser that is opinionated about what fields are relevant to code generation. This produces an easier-to-use type-safe AST, which allows us to separate certain kinds of logic (such as handling unions) into the AST parser rather than in the generators themselves.
• Namer: QuickType did a fairly good job of handling name collisions in events and properties, though we did hit some quirks, such as not handling case-insensitive name collisions for files. We now use our own namer for handling name collisions.

With all of this functionality together, adding a new language looks like this:

1. Build the generator by implementing the various Generator methods
2. Add an example app to tests/e2e that fires off the standard suite of events (see suite.test.ts)
3. Document it! Add it to the README + the init command + Segment docs

Fixes / New Features

We've also resolved a handful of other issues we've encountered with typewriter, such as:

• Better sane defaults, such as using shared analytics clients by default (window.analytics, iOS shared analytics instance, etc.). This means some languages require zero configuration.
• A lack of visibility into typewriter usage, due to a lack of analytics.
• Support for JS Proxies, to avoid program crashes caused by missing analytics methods (f.e., if an event is removed from the Tracking Plan and someone mistakenly syncs it). When this happens, an Unknown Analytics Event Fired event will fire that allows teams to alert on this.
• Support for tree-shaking in JS/TS. Resolves Support for Named Exports #41

Backwards Compatibility

We haven't ported Android support to this version yet, but will be shipping it soon. For now, we recommend using the Android clients generated by typewriter@6.1.5.

The underlying generation logic has heavily changed, so expect the clients to be nearly, but not entirely, compatible with each other. It'll depend on what kind of generated functionality you are using.

Notes

For more context, see: https://paper.dropbox.com/doc/typewriter-Typewriter-2.0-Vision--AnFffUqElJQQOUHtVLGaWnzbAg-FtUl1rmfXyuoVcI8uS5yM

Future Work

v7.0.x

• CI for analytics-ios w/ Swift
• android-java support + CI

v7.x.0

• context.protocols.event_version support
• Identify and Group call support
• Common JSON Schema support
• Per-event and per-label typewriter update support
• api.ts and cli/ test suite
• Run-time JSON Schema validation in iOS + Android
• Typewriter Contributing Docs

[colinking]

Alright, we still need to add support for Android and I'd like to hook up the Swift e2e example to CI, but otherwise we're ready to release this.

I'm going to go ahead and merge what we have since this PR has been around for long enough. For folks who need Android support for now, y'all will just want to use v6.