New year, new offsite! This time, I had a solution: ChatGPT; and I was looking for a problem. We built a support bot to answer community questions - which has worked out surprisingly well.

Also, lots of tweaking around with the infrastructure, I've learned different ways pgbouncer can break. I wrote a blogpost about everything we learned trying to make flags reliable: PostHog/posthog.com#5546

More feature flag optimisations, my favourite being with cohorts: PostHog/posthog#14272 - now, instead of relying on computed cohorts to evaluate flags, we convert cohorts into their definitions, and then use those in flags, so it uses the latest properties, irrespective of whether they are in the cohort yet or not.

This speeds things up as well, as we only need to go in one place to evaluate flags (the person properties database).

Focus is back on feature flags, since they are the core of both experiments and features. We are mature enough with respect to features that a decent amount of people have started using them. This has brought up a new problem: Feature flags going down is a no-no. Not only does it affect our customers; but also our customers' customers.

So, make sure our flags are as resilient as possible: PostHog/posthog#13601.

Also, caching is hard.

It's been a while since experimentation released, and we've learned what all mistakes we made. One big mistake I made was not allowing users to change experiment goals once the experiment started. I was worried about p-hacking, but more often than not, the issue was that people made a mistake setting up the experiment, and it was very annoying to restart the experiment just to change the goal, since you're losing all the data you've collected so far.

This month was all about fixing up these mistakes, polishing experiments up so we can learn the new ways things break.

We also spent some time thinking about the long term vision for our team: PostHog/posthog.com#4516

More feature flag improvements. A big one was bootstrapping, which brings local evaluation to the client-side (in a way): PostHog/posthog-js-lite#24. You can initialise your libraries on the frontend with feature flags, which makes them instantly available.

This month was about tackling the server-side libraries feature flag issues. I created a spec that simplified our work, since there was a standard document to replicate across libraries: PostHog/posthog#10459 (comment). This helped us review PRs quickly, and also helped us be faster for implementing, since you could blindly copy the structure, as long as you keep in mind language specific gotchas.

The focus is coming back to feature flags. Specifically, there are a few problems that make them unusable on server-side libraries, and also for user flows (like authentication) where the user identity changes.

It was tricky figuring out how to add support for something like this, while not destroying performance, since this is a very sensitive endpoint. I ended up cleaning a lot of cruft on this endpoint to make this fast.

An explanation of the problem: PostHog/posthog#9547 The solution: PostHog/posthog#10196

The next big thing to tackle is person on events: We are revamping all our queries to become scalable for billion event querying. This requires getting rid of joins as much as possible, and changing our data model a bit to support that. Here's a big overview of this: PostHog/posthog#9802 - this turned out to be a pretty massive refactor.

The new project is support for complex filters in cohorts: not only can we freely combine them, but also added support for complex behaviour like figuring out if a user did an event multiple times / in a specific sequence to become part of the cohort. Github thinks this was my highlight PR for the project: PostHog/posthog#9462

Lots of polishing on Experiments, the last 20% polishing takes surprisingly long. I've come to realise that frontend work takes a lot more time and effort.

We also had our offsite hackathon, out of which came Universal Search: The top searchbar on posthog now searches through everything to give a response.

Automated Insights is the new hardd project, and I'm having a lot of fun remixing ideas: PostHog/posthog#8261. OpenAI is amazing: what would've taken me days to setup 3 years ago now took 30 minutes. 🤯. This is great for quickly validating & discarding ideas.

This month has been amazing. Experimentation looks great, has kicked off well, users (atleast the ones we interviewed) love most of it, and we launched it to everyone on Cloud this week!

This project is different from Correlation Analysis, in the sense that it takes longer for feedback to arrive: Users have to actually finish running their experiments to give feedback on the later-half of the product: the experience while running & on ending an experiment. So, decided to pause improvement work on this for now, setup metrics & docs, and wait for users to use it, listen in to feedback, and then decide how to solve the problems that come up.

Super happy with all that got done this month (it's a LOT, with lots of ...experimenting):

We hit a slight snag with Experimentation: Most users were away during Christmas, which made it hard to schedule feedback calls, which made it hard to iterate quickly. I switched strategies a bit, and now focusing on building out things we definitely need for Experimentation to be successful, sans feedback. Would've loved to validate these, but eh. Will do it as intervews start pouring in.

Went on a 2 week break myself as well, which was refreshing.

Aaaanndd I'm back with a new Project! This time, we're targeting experimentation: PostHog/posthog#7418.

Just like last time, we want to get something working out this sprint. Ofcourse, I started with learning about what existing A/B test platforms do, building a model of why they do things that way, and then tweaking that model so it sits on top of PostHog's existing systems. That's what you see happen in the above thread^.

PostHog/posthog#7462

Support Hero time again. Probably not surprising, but this is becoming more intense: there's lots more people in Slack asking questions, which means I'm stretched thin, but oh, such a nice problem to have.

Guess which week was Support Hero:

After the huge project, it's vacation time! Need time to recharge - it was intense, too many open loops that kept me thinking about work after work.

I shipped Quant Analysis! While last time was just the MVP, this time we polished things up, iterated on user feedback, and went back to them to see if they were happy (yes, they were). This was absolutely brilliant.

I used to think the most complicated part of building a product was the software engineering. Now I think it's figuring out what to build.

Wow, what a sprint! I finally started implementation on the huge-ass Quantitative Analysis Project. What's unique about this project is that I'm leading one for the first time. It started off like this: PostHog/posthog#5543

I was working closely with one other person, and we got an MVP out in week 1. This was faaaast! It allowed us to iterate quickly, gather feedback, and fix data issues quickly. With a product that depends on data being good, this was critical.

I wouldn't link specific PRs, because there were a shit-tonne

Interestingly, I spent a lot more time thinking about the problems to solve here, even after work (which didn't bode well for sleeping peacefully). At the same time, this allowed for some cool technical breakthroughs, where we could run correlations much quicker.

Finally, doing live user interviews was fun - getting feedback from actual users, reading between the lines, figuring out what they want, and then putting those ideas together was a fun challenge: PostHog/posthog#6474. The next sprint is going to be scary ambitious.

Two weeks of working on Paths. We shored up the API, and I finally got my hands on the frontend code. Kea is an amazing library to work with.

One very interesting problem: Validating Graph Edges on Paths: PostHog/posthog#6041 (and linked PRs!)

• Connect Persons on a Path to an API request: PostHog/posthog#6035, PostHog/posthog#6070
• API additions: PostHog/posthog#6124, PostHog/posthog#6111, PostHog/posthog#6052
• Query Optimisations: PostHog/posthog#6125
• Random Frontend stuff: PostHog/posthog#6175,

A (missed) Offsite, Support Hero, vacation to Mallorca <3

I owned my first project! PostHog/posthog#5543

Apart from thinking hard about this, the regular sprint continues. Working on Paths - the new feature!

PostHog/posthog#5646

This week's been pretty cool, because I'm finally doing more of the ground-up startupy stuff: thinking through things from scratch, building PoCs, gathering results, and then finally building the product.

Final sprint for Funnels, got the vaccine, got sick afterwards, didn't do much, except this one big bug fix for funnel breakdowns: PostHog/posthog#5655, PostHog/posthog#5538

Remember how Support Hero was a lot of fun? (Week of 21 May, 2021). Time for round 2. A lot less overwhelming, as I knew a lot more about things (but still not everything).

Random bug fixes: PostHog/posthog#5486 etc. etc.

More bug fixes, some tricky things to grasp, and finally dipped into other unknown areas. I follow a land-and-expand strategy: get really good at understanding one part of the system, then slowly expand from that "base" to understand rest of the system. This usually means that my work speed slows down, as I parse through all the new stuff.

A good way to do this is to pick up bugs at the edges of what you know. That's what I've been doing:

• Dashboard bug: PostHog/posthog#5395
• Caching bugs: PostHog/posthog#5399
• Breakdown in funnels bug: mix of frontend and backend issues - PostHog/posthog#5357

Final sprint before Funnels meant lots of QA, lots of bugfixing, and lots of testing :) - I'm so tired now.

• Special Bugs: When your new technologies don't work like you'd expect: ClickHouse/ClickHouse#26580, and patches for it: PostHog/posthog#5283
• Bug fixes: PostHog/posthog#5174, PostHog/posthog#5277, PostHog/posthog#5292, PostHog/posthog#5308, PostHog/posthog#5315, PostHog/posthog#5316

Lots of gathering requirements, getting to the bottom of new features we want to implement: PostHog/posthog#5074 - and reminders to think from first principles.

• And then doing it: PostHog/posthog#5104
• Some code rearchitectures that make future-work so much easier: PostHog/posthog#5043
• Random bug fixes: PostHog/posthog#5055

Who knew playing around with SQL, and generating interesting queries could be so much fun? This week was more dakka: more add ons, more functionality to the basic funnel APIs. Some clever refactoring + testing mechanisms, that I enjoyed setting up

• Interesting test infra setup with Funnel Breakdowns - PostHog/posthog#5043
• Refactoring to remove obsolete concepts - PostHog/posthog#5037
• Bells and whistles - PostHog/posthog#5024
• Random bug fixes: PostHog/posthog#5055

Some big funnel improvements

• Remember to create tests for backwards compatibility! - PostHog/posthog#4946
• Unordered funnels, people, and more - PostHog/posthog#4943, PostHog/posthog#4890

I was getting pretty comfortable with my role, and that seemed like the best time to switch teams 😂. Purely co-incidental, we shifted focus, and I've been writing wonderful SQL this week. Damn, this is SO MUCH FUN. This week (and hopefully the coming few weeks, really want to brush up on my querying skills). This has been very helpful: https://pgexercises.com/

• Some complex funnels: PostHog/posthog#4868, PostHog/posthog#4863
• When revisiting setting up, always update docs for whoever comes after you :) - PostHog/posthog.com#1506

Finishing up new processes for the Plugin Developer Experience, plus excellent docs.

• PostHog/posthog-plugin-starter-kit#7, PostHog/posthog.com#1467
• Fun bug fixes: PostHog/posthog#4772
• Odds and ends: PostHog/posthog#4755, PostHog/posthog.com#1497

Well, plugin installation is deprioritized for now. New focus: plugin development experience! Lots of time spent thinking about how the documentation should look like, what workflows should the code promote, and what feels confusing.

• Making the Plugin Development Experience nicer: PostHog/posthog.com#1467, PostHog/posthog#4688
• Wrangling with BigQuery: PostHog/bigquery-plugin#9
  • Good Habit of mind - when things are hard to debug, write code to make it easier to debug similar issues in the future: PostHog/plugin-server#465

Thinking about a big project, and learning enough about the interacting systems to design a decent solution can be hard! Really looking forward to finishing the plugin installation step.

• Plugin Install Step: https://github.com/PostHog/plugin-server/issues/405, PostHog/plugin-server#456
• Debugging S3 Queues: PostHog/plugin-server#451

Getting comfortable with the codebase, starting to focus on reviewing others' code. It's interesting to try and model how new changes would affect the existing code. Further, this helped uncover my blindspots - glad I started this earlier than later!

• Some interesting bug fixes: PostHog/posthog-js#233, PostHog/posthog-js#234
  • Accompanying blog post:
• Random odds and ends: PostHog/plugin-server#441, PostHog/plugin-server#447, PostHog/posthog-js#236

I was Support Hero this week! It's... intense! Lots of user issues that I first have to learn about myself, and then solve. This took a surprisingly long amount of time, but was very worth it: it helped me see where actual users of PostHog get stuck.

• Built Plugin Capabilities: PostHog/plugin-server#384, PostHog/posthog#4371
  • This was my first big feature: involving touching a lot of things and understanding the system, so I could come up with a decent solution. Fun stuff!
• Support fixes: PostHog/posthog-python#32

It's funny how this appears to be the least productive week so far, but I felt I got much more out of it, vs the past 2 weeks. I ought to do Support Hero more often than the usual schedule, if possible.

• Disallow Plugins from changing the teamID: PostHog/plugin-server#381
  • I messed up here a bit, forgot to take care of batch events (PostHog/plugin-server#396)
• Add Sentry+Django integration to Python library: PostHog/posthog-python#13
• Bug fixes: feature flag rollout %: PostHog/posthog-python#30
• Random typos and fixes: PostHog/posthog#4315, PostHog/posthog.com#1335

Week 1 at PostHog!

• Build a new plugin - the Downsampling Plugin: PostHog/downsampling-plugin#1
• Add support for $set and $set_once to Python library: PostHog/posthog-python#23
  • I messed up here a bit: PostHog/posthog-python#28 - be moaar careful about tests that don't pass on local but pass on master
  • Interesting difference in workflow causing bugs: I didn't think of pulling master because I'm used to working off of forks, vs pre-existing branches
• Random typos and fixes: PostHog/posthog#4220, PostHog/posthog.com#1307, PostHog/posthog.com#1316