{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":707423360,"defaultBranch":"master","name":"ragnar","ownerLogin":"killthebuddh4","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-10-19T21:42:24.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/100799178?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1711059460.0","currentOid":""},"activityList":{"items":[{"before":"f9a3590d02a7aee1dc190b270b6dc0755193305e","after":"d48646eb376934879d3f6f4f79ed69d6ea89893d","ref":"refs/heads/master","pushedAt":"2024-03-29T19:03:41.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"killthebuddh4","name":"achilles schmelzer","path":"/killthebuddh4","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/100799178?s=80&v=4"},"commit":{"message":"Add embeddings and search endpoint\n\nThe rate limits for the OpenAI embeddings API are pretty easy to work\nwith, so the /embed endpoint is straightforward.\n\nRight now we just save the ~30MB of embeddings into a JSON file and load\nthe whole file at startup. Looping through the 10K-ish embeddings is not\na problem at all for an educational project.\n\nI might add a proper vector store later just to show how it's done, but\nI think the raw cosine with a JSON file is more demonstrative of what\nembeddings \"are\".\n\nThe chunk, embed, and search endpoints all work, but it's not clear at\nall that they're working correctly. This is one of the ideas I want to\nillustrate: to get something that \"works\", you need to tune things for\nyour use case. You need to clean your data and set a number of\nparameters to values that work for you.\n\nThe next thing I'm going to do is sift a little bit through the data and\nAPI and see if I can't get the search to work a little bit nicely. That\nsaid, I need to be careful about making too many changes because part of\nthe point of the basic implementation is to show that even an\nimplementation without any \"bugs\" may be unuseable.","shortMessageHtmlLink":"Add embeddings and search endpoint"}},{"before":"e63e5ce38cfbadaf31c1eee5fd36cf962867a442","after":"f9a3590d02a7aee1dc190b270b6dc0755193305e","ref":"refs/heads/master","pushedAt":"2024-03-25T19:47:12.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"killthebuddh4","name":"achilles schmelzer","path":"/killthebuddh4","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/100799178?s=80&v=4"},"commit":{"message":"Add basic /chunks endpoint\n\n- Added tokenizer library\n\nChunking is pretty slow but is also just about as naive as possible.\nSome low-hanging fruit:\n\n- parallel tokenization\n- binary search for mapping text to tokens\n- batch inserts\n\nSome things I need to do next:\n\n- Add zod schemas corresponding to database tables\n- Add embedding route","shortMessageHtmlLink":"Add basic /chunks endpoint"}},{"before":"3b8580164eecf8fa621243814fc9d9bc2cc0296d","after":"e63e5ce38cfbadaf31c1eee5fd36cf962867a442","ref":"refs/heads/master","pushedAt":"2024-03-22T22:55:57.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"killthebuddh4","name":"achilles schmelzer","path":"/killthebuddh4","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/100799178?s=80&v=4"},"commit":{"message":"Add express and supabase\n\n- Add a files table\n- Add a /load route\n- Add a /stats route\n- Add tests for routes\n\nThe next few things I need to do (not necessarily in order)\n\n- Implement /etl/basic\n- Start the README\n- Add the vector plugin to the supabase dev setup","shortMessageHtmlLink":"Add express and supabase"}},{"before":null,"after":"3b8580164eecf8fa621243814fc9d9bc2cc0296d","ref":"refs/heads/master","pushedAt":"2024-03-21T22:17:40.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"killthebuddh4","name":"achilles schmelzer","path":"/killthebuddh4","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/100799178?s=80&v=4"},"commit":{"message":"Init commit","shortMessageHtmlLink":"Init commit"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEIujnuwA","startCursor":null,"endCursor":null}},"title":"Activity · killthebuddh4/ragnar"}