Skip to content

Commit

Permalink
Reduce complexity for cd test for now (#662)
Browse files Browse the repository at this point in the history
  • Loading branch information
jobergum committed Jan 17, 2024
1 parent 0d90fa5 commit de5e709
Showing 1 changed file with 7 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,9 @@
"and feed it to a Vespa instance. The dataset comprises of English language arXiv papers from the Cornell/arXiv dataset, with two new columns added: title-embeddings and abstract-embeddings. Embeddings generated using the [bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) embeddings model. \n",
"\n",
"In this notebook, we use Vespa's embedder functionality to include the [bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) embedding\n",
"model into Vespa for query serving. "
"model into Vespa for query serving. \n",
"\n",
"This is work in progress - we want to demonstrate more query examples. "
]
},
{
Expand Down Expand Up @@ -432,7 +434,7 @@
"# app:Vespa = vespa_cloud.deploy()\n",
"\n",
"from datasets import load_dataset\n",
"dataset = load_dataset(\"somewheresystems/dataclysm-arxiv\", split=\"train\", streaming=True)\n",
"dataset = load_dataset(\"somewheresystems/dataclysm-arxiv\", split=\"train\", streaming=True).take(100)\n",
"vespa_feed = dataset.map(lambda x: \n",
"{\n",
" \"id\": x[\"id\"],\n",
Expand All @@ -454,8 +456,10 @@
"def callback(response:VespaResponse, id:str):\n",
" if not response.is_successful():\n",
" print(f\"Document {id} failed to feed with status code {response.status_code}, url={response.url} response={response.json}\")\n",
" else:\n",
" print(f\"Document {id} success.\")\n",
"\n",
"app.feed_iterable(schema=\"paper\", iter=vespa_feed, callback=callback, max_connections=12, max_workers=64, max_queue_size=10000)\n"
"app.feed_iterable(schema=\"paper\", iter=vespa_feed, callback=callback, max_connections=12, max_workers=14, max_queue_size=10000)\n"
]
},
{
Expand Down

0 comments on commit de5e709

Please sign in to comment.