Blog - How we built a scalable, arbitrary code app server (from MVP to millions of events)

[ivanagas]

Strapline

Explain the idea in a sentence or two

How we built a scalable, arbitrary code app server (from MVP to millions of events)

Why should we do it?

How it will it be useful or interesting for readers/viewers

I think it is very interesting, a solid amount of work has already been done by Marius, Yakko, Michael, and could be very popular with our target demographic. Lots there to do a technical deep dive.

#1955

https://www.youtube.com/watch?v=3_yH24Bh0HE

Was talked about but never finished: https://github.com/PostHog/product-internal/issues/152#issuecomment-906544963

Outline

Bullet point outline of structure / questions / topics to be covered

• We wanted the ability for anyone to run code that modifies their event
  • Because PostHog is product analytics, there are infinite use cases of companies who could use us, each with special data requirements, we might not solve them all.
  • To make sure we are capturing as much data as possible for users, we can integrate with other services.
• The beginning
  • Hackathon, adding a world map
  • Failed beginnings
    • Python
  • MVP built in 3 days
• App structure summary
  • Each app has two files: index.js (or index.ts) and plugin.json
    • index contains logic
    • plugin contains configuration for user input
  • Also includes Attachment, Storage, and LogEntry
    • Aka extensions
  • Within index, have functions like processEvent and onEvent which take a single event and the meta object (configuration values basically)
  • Exports a function for use in a VM
• Integrating with PostHog, focusing on areas most relevant to apps.
  • https://posthog.com/docs/how-posthog-works/ingestion-pipeline
  • Need to maintain scalability (millions of events+)
  • Integrating with our ingestion pipeline
    • Redis (minor importance)
      • Can cache data for the app: https://posthog.com/docs/apps/build/reference#cache
      • Redlock for scheduler, job queue consumer
        • Guarantee that it only runs once
        • For example, you only need one scheduler, if you have more than one then tasks will run more than once
      • Send PubSub Redis Reload
        • Plugins have changed
    • Kafka (minor importance)
      • For data flows, batches, split, work in parallel on plugin server
      • Batching events: Kafka last mile batching & retrying processEvent plugin-server#273
    • Sending processed data to events_plugin_ingestion Kafka topic
  • Plugin server, focus on areas most relevant to apps.
    • Main thread (control plane)
      • Takes tasks, starts services
      • Manage threads with Piscina
        • Abstracts a lot away
      • Scheduler
      • Job queue
    • Worker threads (receive and do tasks)
      • Managed by Piscina
      • Each thread can run up to 10 tasks at the same time
      • Structured with ingestion logic, connections object, and VMs (see App structure above)
      • Tasks relevant to apps
        • Apps - processEvent
          • https://github.com/PostHog/posthog-plugin-geoip
        • PostHog processing
          • Person process
          • Event processing
          • Writing to Clickhouse
        • Apps - onEvent
          • Export, alert apps
• Building more functionality
  • Scheduling: Add support for scheduled plugins plugin-server#15
  • Export events
  • Statelessness
    • No need for team specific context: Stateless plugins posthog#8112
  • Reliability and mangement improvements: Plugin Server - Before and After posthog#8808
    • Event ingestion slowdown under high load posthog#8477
• How did we make sure this doesn’t break things?
  • Allowlisted libraries
    • https://posthog.com/docs/apps/build/reference#available-packages-and-imports
    • aka Imports
  • Breaking for loops
    • Babel Loop Timeouts Babel Loop Timeouts plugin-server#155
      • If loop has been running for more than 30 seconds, error
  • Review
    • Self-hosted you can do whatever you want
    • Cloud are reviewed before publishing
  • Templates and testing
    • We provide templates
    • Allow the use of Jest for testing
  • RetryError
    • https://posthog.com/docs/apps/build/reference#maximizing-reliability-with-retryerror
    • Limited retries before failing/ignoring
  • Scheduler
    • Runs every minute, if the last event is done
  • Typescript
• What does this allow
  • You can create and upload your own app.
    • Many plugins have been created by our community
    • Large customers writing their own apps for their needs
  • Anyone can run what they want when self-hosting.
  • Expand functionality of PostHog, allow people to customize their injestion and data processing themselves
• Summary
  • Build an MVP, limit scope
  • Integrate with what you have and what is most useful for users
  • Improve robustness and stability
  • Expand functionality

[andyvan-ph]

Go for it. This has Hackernews all over it