PostgreSQL RAG Spike

Using PostgreSQL with the new pgvector extension, enabling vector search.

As I prefer using MySQL, it shouldn't be too much effort to replace MySQL with PostgreSQL. And it's going to be easier to support one database than many.

Architecture

• Express API is the backbone holding everything together
• PostgreSQL database for persisting embeddings as well as the normal data used by Express API (instead of MySQL)
• FastAPI wrapper for embeddings model. It turns out we can't just use library from npm for this.

Spinning up the local environment

Spin up the environment

docker compose -p postgresql-rag up -d

SSH into the container

docker compose -p postgresql-rag exec api bash

Access the database via psql client

psql -U root -h postgresql_rag

Rebuild the database

npm run rebuild-db

Spin down the environment

docker compose -p postgresql-rag down

PostgreSQL Cheatsheet

List databases

\l

Connect to database

\c postgresql_rag

List tables

\dt

Articles

/articles

Okay, this page is the finished, polished version, as it were.

• Upload article which is persisted in articles table
• Article is split into sentences using compromise.js and then turned into embedding vector using HuggingFace Sentence model via RESTful API
• Embeddings are persisted into articles_embeddings table, linking back to article
• Search is turned into embedding vector and returns matching articles via articles_embeddings similarity and threshold (debugging tool)
• Prompt works similarly to search but inserts the matching articles into the context for the prompt before sending it off to LLM and returning results

Next steps

• In a separate project / repo, build the AI chatbot as a SaaS product using learnings from this spike

Example generating content for knowledge table

Here, we're making RESTful API POST to API. The API will turn the description into an embedding and insert it into the knowledge table. At least, that's the plan!

 curl -X POST -H 'Content-Type: application/json' -d '{"description":"Optimizing your website with relevant keywords and high-quality content is crucial for improving your SEO and ranking higher in search engine results."}' http://localhost:8282/api/knowledge

Example Vector search queries

Let's generate a search embedding for capturing embeddings that are close enough - based on seed data

curl -X POST -H "Content-Type: application/json" -d '{"sentence": "What can you tell me about Vietnam?"}' http://localhost:7272/api/generate-embedding

Let's try searching the database for similar descriptions via API

curl -X POST -H "Content-Type: application/json" -d '{"search": "What can you tell me about SEO?"}' http://localhost:8282/api/search

SELECT
  id,
  description,
  1 - (embedding <=> ARRAY[-0.01554377656430006,0.022419974207878113,0.014746277593076229,-0.06287446618080139,0.0013057268224656582,0.0060835243202745914,-0.0760607197880745,-0.0025474673602730036,0.009897692129015923,-0.05662404000759125,0.007740023545920849,-0.0051863365806639194,-0.0015972841065376997,-0.038047123700380325,-0.036658644676208496,-0.017571840435266495,0.03276313468813896,-0.054045338183641434,-0.049213919788599014,-0.022915195673704147,-0.03462139144539833,-0.03266045078635216,-0.003020461881533265,-0.051200009882450104,0.047195978462696075,-0.02434905432164669,-0.07450001686811447,0.005315870046615601,-0.01933848112821579,-0.041156042367219925,-0.0025186778511852026,-0.017208613455295563,-0.016353297978639603,-0.0027180835604667664,1.4431155932470574e-06,-0.0033118166029453278,0.020741427317261696,-0.0010861627524718642,-0.01163704413920641,0.031437523663043976,0.09824078530073166,-0.05731803923845291,0.019678322598338127,-0.016956746578216553,0.03388991951942444,-0.11209940910339355,0.04720096290111542,0.03754914179444313,0.037563174962997437,-0.00038891154690645635,0.003396342508494854,0.053899720311164856,-0.0838850662112236,-0.031173422932624817,0.014758285135030746,0.035484082996845245,0.036393459886312485,0.045549746602773666,0.004395903088152409,-0.0707777589559555,-0.04403924196958542,0.00784715823829174,-0.0695975124835968,0.008193415589630604,-0.007822425104677677,0.041230835020542145,0.02282528579235077,0.03177745267748833,-0.009097359143197536,0.00041502012754790485,-0.02296694926917553,-0.030449768528342247,-0.025070784613490105,0.07891130447387695,0.0013345424085855484,-0.048014674335718155,0.004903214517980814,0.00354012125171721,0.03570851311087608,0.029249435290694237,0.05332915484905243,-0.01274979766458273,0.03880718722939491,-0.0050177196972072124,0.04183526337146759,-0.07083927094936371,-0.019953563809394836,-0.017096398398280144,-0.06925591081380844,-0.04110928997397423,-0.01395423524081707,0.0291073527187109,-0.01868300884962082,0.06291419267654419,-0.024534959346055984,-0.03922685235738754,0.01418889407068491,0.020840393379330635,0.014342171140015125,-0.0659644827246666,-0.05680975317955017,-0.0049676974304020405,-0.07904719561338425,-0.010722377337515354,-0.0049079046584665775,0.06319639831781387,0.009615297429263592,-0.012095958925783634,-0.017232432961463928,-0.042532797902822495,-0.07532983273267746,0.011439185589551926,0.015205129981040955,0.05701368674635887,0.058797821402549744,-0.07085513323545456,0.034812092781066895,0.025194810703396797,0.04013784974813461,0.05103697255253792,-0.014234624803066254,-0.03312654793262482,-0.008737616240978241,0.03587113693356514,-0.014257953502237797,0.04382670670747757,-0.01072827447205782,0.017004648223519325,-0.0015338440425693989,0.02006230503320694,0.008608425036072731,-0.010915976949036121,-0.019572664052248,0.015962358564138412,-0.005497320089489222,0.04715960472822189,-0.008592063561081886,0.08376994729042053,-0.040821872651576996,-0.0045270053669810295,0.021064795553684235,-0.04633955657482147,0.01929529383778572,-0.007001532707363367,0.02416778914630413,0.044748224318027496,0.03463919460773468,-0.008944539353251457,0.0008215727866627276,-0.05799492448568344,-0.018634427338838577,0.0509406141936779,-0.033451445400714874,-0.023592431098222733,0.02384874038398266,0.02795664593577385,-0.030310960486531258,0.011568141169846058,-0.0016703951405361295,0.019797930493950844,0.04103975370526314,0.0382225327193737,0.016241535544395447,-0.08839438855648041,-0.024996977299451828,-0.01747547648847103,-0.010895361192524433,-0.015642043203115463,-0.06929176300764084,-0.0026508518494665623,0.048589494079351425,0.009289370849728584,0.02297830954194069,-0.028693554922938347,-0.01038889680057764,-0.0742688998579979,0.007141636218875647,-0.04590509086847305,0.005070860497653484,0.029885806143283844,0.054135214537382126,-0.021979080513119698,0.0032798179890960455,-0.05158877372741699,0.010527560487389565,0.06615598499774933,0.016080379486083984,0.012281321920454502,-0.008952143602073193,-0.023406196385622025,0.0025428521912544966,-0.03880209103226662,0.03398966044187546,0.007898010313510895,-0.005055538844317198,0.005962014198303223,0.012953103519976139,-0.009398682974278927,0.047678567469120026,0.02629147656261921,-0.04725765064358711,0.040823325514793396,-0.019777124747633934,0.017950918525457382,0.035751666873693466,-0.013390414416790009,0.01699739322066307,-0.03074730932712555,0.0017258741427212954,-0.02797507867217064,0.050749946385622025,0.011947295628488064,0.07469896972179413,0.03220188617706299,0.027826916426420212,0.016005542129278183,0.017103563994169235,-0.025668758898973465,0.029243864119052887,0.03556938096880913,-0.01788252592086792,0.011797571554780006,-0.02102375216782093,-0.02640709839761257,0.07492518424987793,-0.03253009915351868,0.0674847960472107,-0.02067500166594982,0.007664153818041086,-0.00900940876454115,0.020824667066335678,0.042131055146455765,-0.00919659435749054,-0.029388120397925377,0.030681747943162918,0.03691153973340988,0.02328375354409218,0.03825727477669716,-0.033169738948345184,-0.08036690950393677,0.007799158338457346,-0.011575094424188137,0.025906767696142197,0.022784700617194176,0.03910554200410843,0.016058383509516716,0.006732016336172819,0.009159042499959469,0.04621218144893646,0.019421296194195747,0.014436936937272549,-0.011041641235351562,0.02199050970375538,-0.03522366285324097,0.01799605041742325,0.05351626127958298,-0.0164585430175066,-0.024935614317655563,0.052222173660993576,-0.017215579748153687,0.010604307986795902,-0.01679290272295475,-0.02442316710948944,-0.04348130151629448,-0.009559271857142448,0.036713019013404846,0.05248185992240906,-0.06418542563915253,0.017763379961252213,0.015485696494579315,0.029867924749851227,0.00021592818666249514,-0.014749648049473763,0.010283881798386574,-0.04982298985123634,-0.008479800075292587,0.0415985994040966,-0.0163898766040802,-0.0031172398012131453,0.03233323618769646,0.043775588274002075,0.06602997332811356,-0.0215061716735363,0.04711225628852844,-0.007600118406116962,0.029267245903611183,-0.003264068393036723,-0.07545550912618637,0.034384988248348236,0.025840716436505318,0.02505437843501568,0.03225010633468628,-0.047070641070604324,-0.08004491031169891,0.038307491689920425,0.007081128656864166,0.030597159639000893,-0.027278272435069084,0.0034173415042459965,0.08638930320739746,-0.004519458394497633,-0.03566226735711098,0.08044198155403137,0.0018291861051693559,-0.08044808357954025,-0.005122707225382328,-0.019338926300406456,-0.04259328916668892,-0.14954936504364014,0.057062987238168716,0.036620743572711945,0.014177709817886353,0.06165090203285217,-0.06319981813430786,0.03254114091396332,0.071773000061512,-0.01069287583231926,0.04245928302407265,0.00717403506860137,-0.03152364119887352,0.01651799865067005,-0.010067668743431568,0.013202075846493244,0.06745442003011703,0.0011477430816739798,-0.0032626003958284855,-0.004517323337495327,-0.03187498450279236,0.06778968125581741,0.03725191950798035,0.009487304836511612,-0.0028887635562568903,0.002646335866302252,0.012857986614108086,0.0034590638242661953,0.002986556151881814,-0.012993858195841312,-0.05805283039808273,-0.04455677419900894,-0.05239911377429962,-0.0186387300491333,0.04006732627749443,-0.007258032448589802,-0.008956009522080421,-0.0488753616809845,0.01145809143781662,-0.049597010016441345,0.02145516499876976,-0.023128364235162735,-0.015529331751167774,0.04866046831011772,-0.02731228992342949,0.01947377808392048,-0.031809672713279724,-0.013732592575252056,0.05959280952811241,-0.04581613093614578,-0.044600650668144226,0.027656314894557,0.019731789827346802,0.010885970667004585,0.019478533416986465,-0.007103913929313421,0.03928523510694504,0.05213366076350212,0.023891830816864967,0.008241899311542511,0.007641815114766359,0.005886556580662727,-0.009344338439404964,-0.0008339640335179865,0.0002693622082006186,-0.005541092716157436,0.03276432305574417,-0.03561234846711159,-0.03701990470290184,0.03414193540811539,0.03443784639239311,0.01992974802851677,0.03039364144206047,0.01974949799478054,-0.05870017036795616,0.01304411981254816,-0.013071662746369839,-0.024538302794098854,-0.009371516294777393,0.04633292555809021,0.02868022210896015,0.005603929981589317,-0.06382796168327332,0.037397392094135284,-0.011883205734193325,-0.028474606573581696,0.05674540996551514,-0.030434254556894302,0.03028700314462185,0.01791493594646454,-0.025433765724301338,0.005007144995033741,-0.05078258365392685,-0.007201267406344414,-0.021267715841531754,-0.004020280204713345,0.06023513153195381,0.00772298825904727,-0.00500325346365571,-0.02263144962489605,-0.032523173838853836,0.01421072892844677,-0.004131638444960117,0.06037648767232895,0.027198048308491707,0.03204716742038727,0.03515521064400673,-0.009814474731683731,-0.00014069357712287456,-0.03736381232738495,-0.013148642145097256,-0.003683574264869094,0.03314534202218056,0.005138983950018883,0.002200559712946415,-0.06362558156251907,-0.07515273988246918,0.010560121387243271,0.039545342326164246,0.019272658973932266,0.042132459580898285,-0.027977710589766502,0.023029427975416183,0.0005612385575659573,0.020922735333442688,-0.023559018969535828,-0.026217855513095856,-0.005274986382573843,-0.01623661071062088,0.03854433447122574,0.02166919969022274,-0.037737391889095306,-0.0022566558327525854,0.035422634333372116,0.030637621879577637,-0.0035457510966807604,0.00958333257585764,0.009526519104838371,-0.03340780735015869,-0.011934311129152775,-0.005371953826397657,0.0593130998313427,-0.026977522298693657,-0.02637624926865101,0.02848811447620392,-0.045633114874362946,-0.009600265882909298,-0.021846750751137733,0.023559125140309334,-0.029376696795225143,-0.04453986510634422,0.031209316104650497,0.08456986397504807,0.006393734831362963,0.028074687346816063,0.03166288882493973,0.0207925196737051,0.058517687022686005,-0.00899286288768053,0.07000253349542618,-0.007165113929659128,-0.037996068596839905,-0.0024544796906411648,0.0057266028597950935,-0.0039760274812579155,-0.031114641577005386,-0.007898147217929363,-0.051518842577934265,-0.047939401119947433,-0.0602533183991909,-0.026100656017661095,-0.01123407669365406,-0.008160634897649288,0.004018103238195181,0.06114013120532036,-0.024809332564473152,-0.040974318981170654,-0.027422690764069557,-0.024551114067435265,-0.08894894272089005,0.007732049096375704,0.030317313969135284,-0.031036756932735443,-0.04754439368844032,-0.03952830284833908,0.014724677428603172,-0.03152519837021828,0.014183848164975643,0.03663200885057449,0.008373359218239784,-0.00038330687675625086,0.029893886297941208,0.038855504244565964,0.03150057792663574,-0.003038438968360424,0.02448083460330963,0.005324238911271095,0.016229860484600067,-0.035176727920770645,-0.007721320725977421,0.0709470883011818,0.014698677696287632,0.011428040452301502,0.058094728738069534,0.006125491578131914,-0.02021559327840805,0.003424493595957756,-0.05513014644384384,-0.0005770211573690176,-0.01358573418110609,-0.0044650789350271225,-0.015521541237831116,-0.051729217171669006,0.011778275482356548,0.03744928538799286,0.03918590396642685,-0.019777854904532433,0.05320800468325615,-0.01124056801199913,-0.0031177501659840345,0.016752881929278374,0.0879531279206276,0.04402780160307884,-0.003838692791759968,-0.006754045374691486,0.07400249689817429,0.025245321914553642,0.002208446152508259,0.071981281042099,0.06698610633611679,0.10805130004882812,-0.0561223104596138,0.01935388334095478,-0.010747314430773258,-0.010381125845015049,0.01167341973632574,0.009279918856918812,0.009634631685912609,0.00971334520727396,0.02231091447174549,-0.00395305035635829,0.02392105758190155,0.024827035143971443,-0.04686257988214493,0.022999320179224014,-0.09854698926210403,-0.0006394678493961692,0.10101891309022903,-5.374565516128908e-33,-0.06348533928394318,-0.03374747186899185,-0.048214830458164215,0.004246674478054047,-0.05455827713012695,-0.03841743618249893,-0.00958634540438652,0.04998006671667099,-0.061593182384967804,-0.007094107568264008,0.03762327879667282,-0.05839256942272186,-0.005951395723968744,0.018999438732862473,0.029537197202444077,-0.0106028001755476,-0.02148163877427578,0.08883341401815414,-0.00993263814598322,0.0019305896712467074,-0.029283475130796432,-0.003407596843317151,-0.006558115594089031,0.022850066423416138,0.0774165615439415,-0.0072538964450359344,-0.02871154248714447,-0.007090768311172724,0.010224289260804653,-0.0038463266100734472,-0.04330524429678917,0.03149888664484024,-0.05866863951086998,0.03010731004178524,-0.00817460473626852,-0.08763188123703003,-0.0669412612915039,0.01568448357284069,0.04515402019023895,0.01592893712222576,-0.014066974632441998,-0.07176808267831802,0.04006483405828476,-0.01122602540999651,0.002909501548856497,0.0047782231122255325,-0.03867853060364723,-0.0438433513045311,-0.027470791712403297,0.006155714392662048,0.012574322521686554,-0.024118496105074883,0.006254726555198431,-0.04532618075609207,-0.02283252403140068,0.040349531918764114,-0.011432012543082237,0.0850367322564125,-0.045787714421749115,-0.023905467242002487,-0.045933157205581665,0.010774422436952591,0.004595770034939051,-0.045152004808187485,0.06982459127902985,-0.004837731830775738,0.037617020308971405,-0.07414775341749191,-0.013880603015422821,0.015934213995933533,-0.01508430577814579,-0.01015540212392807,-0.012746092863380909,0.020158031955361366,0.019159430637955666,-0.022676551714539528,0.04815050587058067,-0.016311462968587875,-0.025257134810090065,0.036794263869524,-0.038044173270463943,0.020032642409205437,0.028315559029579163,0.054247040301561356,0.01399184949696064,-0.011457985267043114,0.011593785136938095,0.0391603522002697,-0.020660480484366417,0.03493127599358559,-0.02863219752907753,0.06588709354400635,-0.00010317415581084788,0.0010059168562293053,0.03555665537714958,-0.01079276017844677,-0.008158811368048191,0.02266520820558071,0.0003791924682445824,-0.006867288611829281,-0.0024894755333662033,0.03726588189601898,-0.04895547032356262,-0.006758132018148899,0.01626358926296234,0.049427375197410583,-0.012829320505261421,0.05287495627999306,-0.0240990798920393,0.0033846087753772736,-0.007761084940284491,-0.0072180056013166904,-0.030504770576953888,-0.031759947538375854,-0.06339284032583237,-0.04343584552407265,0.007762798108160496,-0.01707880012691021,-0.015686439350247383,-0.021852849051356316,0.0204913392663002,-0.05462212860584259,-0.0037093698047101498,-0.017747072502970695,-0.023164942860603333,0.037320010364055634,0.0003125436487607658,0.054297320544719696,-0.03204786777496338,0.0486457496881485,0.003730440279468894,0.010150778107345104,2.086041064330857e-07,0.011927397921681404,-0.016811227425932884,0.09130528569221497,-0.010584166273474693,-0.029936455190181732,0.018861526623368263,-0.06856387108564377,-0.022459473460912704,0.003090629819780588,-0.009500035084784031,0.026023922488093376,0.022766664624214172,-0.0059986598789691925,-0.02093595266342163,-0.05342553183436394,-0.02901114523410797,-0.00015875893586780876,-0.08310762047767639,-0.006425846368074417,-0.008026300929486752,0.0820871964097023,0.05494074523448944,-0.03509020432829857,-0.029172275215387344,0.01791914738714695,0.0019893967546522617,-0.029903316870331764,0.007401364855468273,-0.06426012516021729,-0.03272852674126625,0.05845868960022926,-0.0172103401273489,-0.03022725135087967,-0.04863828420639038,0.005216303281486034,-0.06812811642885208,0.0050640166737139225,-0.00028712552739307284,0.033349841833114624,-0.008811874315142632,-0.046623069792985916,-0.07271374017000198,-0.009538055397570133,-0.0235754307359457,0.04036684334278107,0.0019942126236855984,-0.04070485755801201,-0.026762885972857475,-0.007798138540238142,-0.0012119254097342491,0.06591890752315521,0.0030818558298051357,-0.014866099692881107,-0.01799248717725277,0.01628568395972252,0.013785259798169136,0.010338589549064636,0.03582846745848656,-0.019497891888022423,-0.0723889172077179,-0.02333374321460724,0.011047391220927238,0.014477538876235485,0.06074299290776253,0.008152110502123833,0.03692908585071564,-0.03188556805253029,7.299546708005335e-35,0.05192539468407631,-0.03826044872403145,-0.04031703248620033,-0.0103183314204216,-0.006878610700368881,-0.04948464408516884,-0.04841644689440727,-0.03579646348953247,-0.019274897873401642,0.06999609619379044,0.00821260642260313]::vector(768)) AS similarity
FROM
  knowledge
  WHERE (1 - (embedding <=> ARRAY[-0.01554377656430006,0.022419974207878113,0.014746277593076229,-0.06287446618080139,0.0013057268224656582,0.0060835243202745914,-0.0760607197880745,-0.0025474673602730036,0.009897692129015923,-0.05662404000759125,0.007740023545920849,-0.0051863365806639194,-0.0015972841065376997,-0.038047123700380325,-0.036658644676208496,-0.017571840435266495,0.03276313468813896,-0.054045338183641434,-0.049213919788599014,-0.022915195673704147,-0.03462139144539833,-0.03266045078635216,-0.003020461881533265,-0.051200009882450104,0.047195978462696075,-0.02434905432164669,-0.07450001686811447,0.005315870046615601,-0.01933848112821579,-0.041156042367219925,-0.0025186778511852026,-0.017208613455295563,-0.016353297978639603,-0.0027180835604667664,1.4431155932470574e-06,-0.0033118166029453278,0.020741427317261696,-0.0010861627524718642,-0.01163704413920641,0.031437523663043976,0.09824078530073166,-0.05731803923845291,0.019678322598338127,-0.016956746578216553,0.03388991951942444,-0.11209940910339355,0.04720096290111542,0.03754914179444313,0.037563174962997437,-0.00038891154690645635,0.003396342508494854,0.053899720311164856,-0.0838850662112236,-0.031173422932624817,0.014758285135030746,0.035484082996845245,0.036393459886312485,0.045549746602773666,0.004395903088152409,-0.0707777589559555,-0.04403924196958542,0.00784715823829174,-0.0695975124835968,0.008193415589630604,-0.007822425104677677,0.041230835020542145,0.02282528579235077,0.03177745267748833,-0.009097359143197536,0.00041502012754790485,-0.02296694926917553,-0.030449768528342247,-0.025070784613490105,0.07891130447387695,0.0013345424085855484,-0.048014674335718155,0.004903214517980814,0.00354012125171721,0.03570851311087608,0.029249435290694237,0.05332915484905243,-0.01274979766458273,0.03880718722939491,-0.0050177196972072124,0.04183526337146759,-0.07083927094936371,-0.019953563809394836,-0.017096398398280144,-0.06925591081380844,-0.04110928997397423,-0.01395423524081707,0.0291073527187109,-0.01868300884962082,0.06291419267654419,-0.024534959346055984,-0.03922685235738754,0.01418889407068491,0.020840393379330635,0.014342171140015125,-0.0659644827246666,-0.05680975317955017,-0.0049676974304020405,-0.07904719561338425,-0.010722377337515354,-0.0049079046584665775,0.06319639831781387,0.009615297429263592,-0.012095958925783634,-0.017232432961463928,-0.042532797902822495,-0.07532983273267746,0.011439185589551926,0.015205129981040955,0.05701368674635887,0.058797821402549744,-0.07085513323545456,0.034812092781066895,0.025194810703396797,0.04013784974813461,0.05103697255253792,-0.014234624803066254,-0.03312654793262482,-0.008737616240978241,0.03587113693356514,-0.014257953502237797,0.04382670670747757,-0.01072827447205782,0.017004648223519325,-0.0015338440425693989,0.02006230503320694,0.008608425036072731,-0.010915976949036121,-0.019572664052248,0.015962358564138412,-0.005497320089489222,0.04715960472822189,-0.008592063561081886,0.08376994729042053,-0.040821872651576996,-0.0045270053669810295,0.021064795553684235,-0.04633955657482147,0.01929529383778572,-0.007001532707363367,0.02416778914630413,0.044748224318027496,0.03463919460773468,-0.008944539353251457,0.0008215727866627276,-0.05799492448568344,-0.018634427338838577,0.0509406141936779,-0.033451445400714874,-0.023592431098222733,0.02384874038398266,0.02795664593577385,-0.030310960486531258,0.011568141169846058,-0.0016703951405361295,0.019797930493950844,0.04103975370526314,0.0382225327193737,0.016241535544395447,-0.08839438855648041,-0.024996977299451828,-0.01747547648847103,-0.010895361192524433,-0.015642043203115463,-0.06929176300764084,-0.0026508518494665623,0.048589494079351425,0.009289370849728584,0.02297830954194069,-0.028693554922938347,-0.01038889680057764,-0.0742688998579979,0.007141636218875647,-0.04590509086847305,0.005070860497653484,0.029885806143283844,0.054135214537382126,-0.021979080513119698,0.0032798179890960455,-0.05158877372741699,0.010527560487389565,0.06615598499774933,0.016080379486083984,0.012281321920454502,-0.008952143602073193,-0.023406196385622025,0.0025428521912544966,-0.03880209103226662,0.03398966044187546,0.007898010313510895,-0.005055538844317198,0.005962014198303223,0.012953103519976139,-0.009398682974278927,0.047678567469120026,0.02629147656261921,-0.04725765064358711,0.040823325514793396,-0.019777124747633934,0.017950918525457382,0.035751666873693466,-0.013390414416790009,0.01699739322066307,-0.03074730932712555,0.0017258741427212954,-0.02797507867217064,0.050749946385622025,0.011947295628488064,0.07469896972179413,0.03220188617706299,0.027826916426420212,0.016005542129278183,0.017103563994169235,-0.025668758898973465,0.029243864119052887,0.03556938096880913,-0.01788252592086792,0.011797571554780006,-0.02102375216782093,-0.02640709839761257,0.07492518424987793,-0.03253009915351868,0.0674847960472107,-0.02067500166594982,0.007664153818041086,-0.00900940876454115,0.020824667066335678,0.042131055146455765,-0.00919659435749054,-0.029388120397925377,0.030681747943162918,0.03691153973340988,0.02328375354409218,0.03825727477669716,-0.033169738948345184,-0.08036690950393677,0.007799158338457346,-0.011575094424188137,0.025906767696142197,0.022784700617194176,0.03910554200410843,0.016058383509516716,0.006732016336172819,0.009159042499959469,0.04621218144893646,0.019421296194195747,0.014436936937272549,-0.011041641235351562,0.02199050970375538,-0.03522366285324097,0.01799605041742325,0.05351626127958298,-0.0164585430175066,-0.024935614317655563,0.052222173660993576,-0.017215579748153687,0.010604307986795902,-0.01679290272295475,-0.02442316710948944,-0.04348130151629448,-0.009559271857142448,0.036713019013404846,0.05248185992240906,-0.06418542563915253,0.017763379961252213,0.015485696494579315,0.029867924749851227,0.00021592818666249514,-0.014749648049473763,0.010283881798386574,-0.04982298985123634,-0.008479800075292587,0.0415985994040966,-0.0163898766040802,-0.0031172398012131453,0.03233323618769646,0.043775588274002075,0.06602997332811356,-0.0215061716735363,0.04711225628852844,-0.007600118406116962,0.029267245903611183,-0.003264068393036723,-0.07545550912618637,0.034384988248348236,0.025840716436505318,0.02505437843501568,0.03225010633468628,-0.047070641070604324,-0.08004491031169891,0.038307491689920425,0.007081128656864166,0.030597159639000893,-0.027278272435069084,0.0034173415042459965,0.08638930320739746,-0.004519458394497633,-0.03566226735711098,0.08044198155403137,0.0018291861051693559,-0.08044808357954025,-0.005122707225382328,-0.019338926300406456,-0.04259328916668892,-0.14954936504364014,0.057062987238168716,0.036620743572711945,0.014177709817886353,0.06165090203285217,-0.06319981813430786,0.03254114091396332,0.071773000061512,-0.01069287583231926,0.04245928302407265,0.00717403506860137,-0.03152364119887352,0.01651799865067005,-0.010067668743431568,0.013202075846493244,0.06745442003011703,0.0011477430816739798,-0.0032626003958284855,-0.004517323337495327,-0.03187498450279236,0.06778968125581741,0.03725191950798035,0.009487304836511612,-0.0028887635562568903,0.002646335866302252,0.012857986614108086,0.0034590638242661953,0.002986556151881814,-0.012993858195841312,-0.05805283039808273,-0.04455677419900894,-0.05239911377429962,-0.0186387300491333,0.04006732627749443,-0.007258032448589802,-0.008956009522080421,-0.0488753616809845,0.01145809143781662,-0.049597010016441345,0.02145516499876976,-0.023128364235162735,-0.015529331751167774,0.04866046831011772,-0.02731228992342949,0.01947377808392048,-0.031809672713279724,-0.013732592575252056,0.05959280952811241,-0.04581613093614578,-0.044600650668144226,0.027656314894557,0.019731789827346802,0.010885970667004585,0.019478533416986465,-0.007103913929313421,0.03928523510694504,0.05213366076350212,0.023891830816864967,0.008241899311542511,0.007641815114766359,0.005886556580662727,-0.009344338439404964,-0.0008339640335179865,0.0002693622082006186,-0.005541092716157436,0.03276432305574417,-0.03561234846711159,-0.03701990470290184,0.03414193540811539,0.03443784639239311,0.01992974802851677,0.03039364144206047,0.01974949799478054,-0.05870017036795616,0.01304411981254816,-0.013071662746369839,-0.024538302794098854,-0.009371516294777393,0.04633292555809021,0.02868022210896015,0.005603929981589317,-0.06382796168327332,0.037397392094135284,-0.011883205734193325,-0.028474606573581696,0.05674540996551514,-0.030434254556894302,0.03028700314462185,0.01791493594646454,-0.025433765724301338,0.005007144995033741,-0.05078258365392685,-0.007201267406344414,-0.021267715841531754,-0.004020280204713345,0.06023513153195381,0.00772298825904727,-0.00500325346365571,-0.02263144962489605,-0.032523173838853836,0.01421072892844677,-0.004131638444960117,0.06037648767232895,0.027198048308491707,0.03204716742038727,0.03515521064400673,-0.009814474731683731,-0.00014069357712287456,-0.03736381232738495,-0.013148642145097256,-0.003683574264869094,0.03314534202218056,0.005138983950018883,0.002200559712946415,-0.06362558156251907,-0.07515273988246918,0.010560121387243271,0.039545342326164246,0.019272658973932266,0.042132459580898285,-0.027977710589766502,0.023029427975416183,0.0005612385575659573,0.020922735333442688,-0.023559018969535828,-0.026217855513095856,-0.005274986382573843,-0.01623661071062088,0.03854433447122574,0.02166919969022274,-0.037737391889095306,-0.0022566558327525854,0.035422634333372116,0.030637621879577637,-0.0035457510966807604,0.00958333257585764,0.009526519104838371,-0.03340780735015869,-0.011934311129152775,-0.005371953826397657,0.0593130998313427,-0.026977522298693657,-0.02637624926865101,0.02848811447620392,-0.045633114874362946,-0.009600265882909298,-0.021846750751137733,0.023559125140309334,-0.029376696795225143,-0.04453986510634422,0.031209316104650497,0.08456986397504807,0.006393734831362963,0.028074687346816063,0.03166288882493973,0.0207925196737051,0.058517687022686005,-0.00899286288768053,0.07000253349542618,-0.007165113929659128,-0.037996068596839905,-0.0024544796906411648,0.0057266028597950935,-0.0039760274812579155,-0.031114641577005386,-0.007898147217929363,-0.051518842577934265,-0.047939401119947433,-0.0602533183991909,-0.026100656017661095,-0.01123407669365406,-0.008160634897649288,0.004018103238195181,0.06114013120532036,-0.024809332564473152,-0.040974318981170654,-0.027422690764069557,-0.024551114067435265,-0.08894894272089005,0.007732049096375704,0.030317313969135284,-0.031036756932735443,-0.04754439368844032,-0.03952830284833908,0.014724677428603172,-0.03152519837021828,0.014183848164975643,0.03663200885057449,0.008373359218239784,-0.00038330687675625086,0.029893886297941208,0.038855504244565964,0.03150057792663574,-0.003038438968360424,0.02448083460330963,0.005324238911271095,0.016229860484600067,-0.035176727920770645,-0.007721320725977421,0.0709470883011818,0.014698677696287632,0.011428040452301502,0.058094728738069534,0.006125491578131914,-0.02021559327840805,0.003424493595957756,-0.05513014644384384,-0.0005770211573690176,-0.01358573418110609,-0.0044650789350271225,-0.015521541237831116,-0.051729217171669006,0.011778275482356548,0.03744928538799286,0.03918590396642685,-0.019777854904532433,0.05320800468325615,-0.01124056801199913,-0.0031177501659840345,0.016752881929278374,0.0879531279206276,0.04402780160307884,-0.003838692791759968,-0.006754045374691486,0.07400249689817429,0.025245321914553642,0.002208446152508259,0.071981281042099,0.06698610633611679,0.10805130004882812,-0.0561223104596138,0.01935388334095478,-0.010747314430773258,-0.010381125845015049,0.01167341973632574,0.009279918856918812,0.009634631685912609,0.00971334520727396,0.02231091447174549,-0.00395305035635829,0.02392105758190155,0.024827035143971443,-0.04686257988214493,0.022999320179224014,-0.09854698926210403,-0.0006394678493961692,0.10101891309022903,-5.374565516128908e-33,-0.06348533928394318,-0.03374747186899185,-0.048214830458164215,0.004246674478054047,-0.05455827713012695,-0.03841743618249893,-0.00958634540438652,0.04998006671667099,-0.061593182384967804,-0.007094107568264008,0.03762327879667282,-0.05839256942272186,-0.005951395723968744,0.018999438732862473,0.029537197202444077,-0.0106028001755476,-0.02148163877427578,0.08883341401815414,-0.00993263814598322,0.0019305896712467074,-0.029283475130796432,-0.003407596843317151,-0.006558115594089031,0.022850066423416138,0.0774165615439415,-0.0072538964450359344,-0.02871154248714447,-0.007090768311172724,0.010224289260804653,-0.0038463266100734472,-0.04330524429678917,0.03149888664484024,-0.05866863951086998,0.03010731004178524,-0.00817460473626852,-0.08763188123703003,-0.0669412612915039,0.01568448357284069,0.04515402019023895,0.01592893712222576,-0.014066974632441998,-0.07176808267831802,0.04006483405828476,-0.01122602540999651,0.002909501548856497,0.0047782231122255325,-0.03867853060364723,-0.0438433513045311,-0.027470791712403297,0.006155714392662048,0.012574322521686554,-0.024118496105074883,0.006254726555198431,-0.04532618075609207,-0.02283252403140068,0.040349531918764114,-0.011432012543082237,0.0850367322564125,-0.045787714421749115,-0.023905467242002487,-0.045933157205581665,0.010774422436952591,0.004595770034939051,-0.045152004808187485,0.06982459127902985,-0.004837731830775738,0.037617020308971405,-0.07414775341749191,-0.013880603015422821,0.015934213995933533,-0.01508430577814579,-0.01015540212392807,-0.012746092863380909,0.020158031955361366,0.019159430637955666,-0.022676551714539528,0.04815050587058067,-0.016311462968587875,-0.025257134810090065,0.036794263869524,-0.038044173270463943,0.020032642409205437,0.028315559029579163,0.054247040301561356,0.01399184949696064,-0.011457985267043114,0.011593785136938095,0.0391603522002697,-0.020660480484366417,0.03493127599358559,-0.02863219752907753,0.06588709354400635,-0.00010317415581084788,0.0010059168562293053,0.03555665537714958,-0.01079276017844677,-0.008158811368048191,0.02266520820558071,0.0003791924682445824,-0.006867288611829281,-0.0024894755333662033,0.03726588189601898,-0.04895547032356262,-0.006758132018148899,0.01626358926296234,0.049427375197410583,-0.012829320505261421,0.05287495627999306,-0.0240990798920393,0.0033846087753772736,-0.007761084940284491,-0.0072180056013166904,-0.030504770576953888,-0.031759947538375854,-0.06339284032583237,-0.04343584552407265,0.007762798108160496,-0.01707880012691021,-0.015686439350247383,-0.021852849051356316,0.0204913392663002,-0.05462212860584259,-0.0037093698047101498,-0.017747072502970695,-0.023164942860603333,0.037320010364055634,0.0003125436487607658,0.054297320544719696,-0.03204786777496338,0.0486457496881485,0.003730440279468894,0.010150778107345104,2.086041064330857e-07,0.011927397921681404,-0.016811227425932884,0.09130528569221497,-0.010584166273474693,-0.029936455190181732,0.018861526623368263,-0.06856387108564377,-0.022459473460912704,0.003090629819780588,-0.009500035084784031,0.026023922488093376,0.022766664624214172,-0.0059986598789691925,-0.02093595266342163,-0.05342553183436394,-0.02901114523410797,-0.00015875893586780876,-0.08310762047767639,-0.006425846368074417,-0.008026300929486752,0.0820871964097023,0.05494074523448944,-0.03509020432829857,-0.029172275215387344,0.01791914738714695,0.0019893967546522617,-0.029903316870331764,0.007401364855468273,-0.06426012516021729,-0.03272852674126625,0.05845868960022926,-0.0172103401273489,-0.03022725135087967,-0.04863828420639038,0.005216303281486034,-0.06812811642885208,0.0050640166737139225,-0.00028712552739307284,0.033349841833114624,-0.008811874315142632,-0.046623069792985916,-0.07271374017000198,-0.009538055397570133,-0.0235754307359457,0.04036684334278107,0.0019942126236855984,-0.04070485755801201,-0.026762885972857475,-0.007798138540238142,-0.0012119254097342491,0.06591890752315521,0.0030818558298051357,-0.014866099692881107,-0.01799248717725277,0.01628568395972252,0.013785259798169136,0.010338589549064636,0.03582846745848656,-0.019497891888022423,-0.0723889172077179,-0.02333374321460724,0.011047391220927238,0.014477538876235485,0.06074299290776253,0.008152110502123833,0.03692908585071564,-0.03188556805253029,7.299546708005335e-35,0.05192539468407631,-0.03826044872403145,-0.04031703248620033,-0.0103183314204216,-0.006878610700368881,-0.04948464408516884,-0.04841644689440727,-0.03579646348953247,-0.019274897873401642,0.06999609619379044,0.00821260642260313]::vector(768)) ) > 0.4
ORDER BY similarity DESC
LIMIT
  10;

TODO

• Insert embeddings via Express API
• Perform similarity search via Express API
• Generate more embeddings and test, test, test
• Given lots of text, split it up into "sentences" and get embeddings and do inserts
• Revisit and optimise the "batch" sentences, including embeddings API
• Create Chatbot on landing page
• Plumb in support for RAG by generating embedding of prompt and sending it all to LLM
• Spike more real-life scenarios like T&Cs, Regulations etc and restrict similarity by keywords or tags (re-modelling req'd)