agent-contextr

What is this?

I was inspired by the system design excercise around writing a web crawler, so I've started doing this for fun.

The idea is that we're going to only pull data from one site at a time, and pull everything down to transform it into a doc that an agent can use.

This project aims to be polite - we respect robots.txt, and if we don't and it's raised that should be fixed immediately. We want to play nicely with the web and the lovely folks on it, not antagonize them.

Setup

To install dependencies:

bun install

To run:

bun run index.ts

This project was created using bun init in bun v1.3.3. Bun is a fast all-in-one JavaScript runtime.

Running the project

Use the run command above, and ensure you drop a few URL's in the seed file provided.