Jev for real-time data pipeline transformation, first impressions

[gitcommitshow]

I spent an afternoon wiring Jev, TypeSafe AI's System 1 decision model, into a data pipeline's in-flight transformation.

The use case

As Paul and Nikita mentioned recently that the bot detection and human verification is the need of the hour. And a simple bot vs human detection is not enough. In the pre-agentic world, it was common to block/ignore the bot traffic but not all bots are the same anymore.

AI agents are the new customers.
Treating all bots the same means blocking customers.

We need a "system 1" (fast and cheap) intelligence to understand and segregate the traffic based on who are these website visitors and what are their intentions. So for every important customer event on the website, like a signup or a purchase, we need to answer two questions before it reaches the warehouse or downstream business/marketing/product/analytics tool.

1. Who produced this customer event? A human or a bot like an AI agent, a search or SEO crawler, or a scraper (classify the specific category of the bot)
2. What is the probability of them having a badIntent?

I got Jev to answer these two questions for each high-stakes event in my RudderStack-powered customer data pipeline, enriching the event data with this additional detail before it reached the warehouse or other tools.

What I learned

It's fast enough to run on real-time event stream

Well, I had tried the same task earlier using the fastest LLMs, and as expected, they were slow to sit inside a data pipeline in-flight transformation, where every event has to be processed in real time and a transformation has only seconds to finish. Also their probability score won't be reliable. On the other hand, Jev answered quickly enough that it never felt like a bottleneck and its probability score is actually usable. It makes "AI inside the pipeline" practical for the intelligent decision making.

It is cheap but is it cheap enough for scale!

My early tests costed me

• $0.0022 for 73 requests (780 tokens each)
• Per decision cost: $0.00003

Which sounds like nothing. But it translates to $30 per million events. For a company sending hundreds of millions of events a day, running it on everything would cost thousands of dollars daily.

So I wouldn't run it on every track or identify event yet. I'd run it where bad actors cost me real money. For signups, logins, and purchases, it's close to free: 10,000 signups a day comes to about 30 cents. I'll let the rule-based filters handle the obvious bots, and save Jev for the decisions that matter the most.

Asking two separate questions looks promising

The most interesting result was a fake signup from a bot farm. Jev said it looked like a human, because it used a real browser, but gave it a high probability of abuse, because of the disposable email and the anonymizing network. A single "is this a bot?" check would likely have let it through.

Accuracy is still an open question

My handful of test events came out right, but not all on the first try. An AI agent completing a checkout for a real customer was first labeled as generic automation, with low confidence. It only got it right after I passed it more context about where the session started. Helpfully, Jev told me it was unsure rather than guessing confidently. But a few test cases prove the idea works, not that it's reliable. I'd want to run it against a much larger set of real, labeled traffic before trusting it with decisions.

My conclusion

Jev is fast enough for in-flight data transformations, promising at not only identifying human vs bot but also distinguishing a bad bot from a helpful AI agent bot. Whether it's accurate enough at scale is the next thing to test. You can help by trying it for your data pipeline and sharing the results in this thread for comparison - https://github.com/gitcommitshow/jev-classifies-bots/