Generative AI Examples v0.8 Release Notes

OPEA Release Notes v0.8

What’s New in OPEA v0.8

• Broaden functionality

  • Support frequently asked questions (FAQs) generation GenAI example
  • Expand the support of LLMs such as Llama3.1 and Qwen2 and support LVMs such as llava
  • Enable end-to-end performance and accuracy benchmarking
  • Support the experimental Agent microservice
  • Support LLM serving on Ray

• Multi-platform support

  • Release the Docker images of GenAI components under OPEA dockerhub and support the deployment with Docker
  • Support cloud-native deployment through Kubernetes manifests and GenAI Microservices Connector (GMC)
  • Enable the experimental authentication and authorization support using JWT tokens
  • Validate ChatQnA on multiple platforms such as Xeon, Gaudi, AIPC, Nvidia, and AWS

• OPEA Docker Hub: https://hub.docker.com/u/opea

Details

GenAIExamples

• ChatQnA

  • Add ChatQnA instructions for AIPC(26d4ff)
  • Adapt Vllm response format (034541)
  • Update tgi version(5f52a1)
  • Update README.md(f9312b)
  • Udpate ChatQnA docker compose for Dataprep Update(335362)
  • [Doc] Add valid micro-service details(e878dc)
  • Updates for running ChatQnA + Conversational UI on Gaudi(89ddec)
  • Fix win PC issues(ba6541)
  • [Doc]Add ChatQnA Flow Chart(97da49)
  • Add guardrails in the ChatQnA pipeline(955159)
  • Fix a minor bug for chatqna in docker-compose(b46ae8)
  • Support vLLM/vLLM-on-Ray/Ray Serve for ChatQnA(631d84)
  • Added ChatQnA example using Qdrant retriever(c74564)
  • Update TEI version v1.5 for better performance(f4b4ac)
  • Update ChatQnA upload feature(598484)
  • Add auto truncate for embedding and rerank(8b6094)

• Deployment

  • Add Kubernetes manifest files for deploying DocSum(831463)
  • Update Kubernetes manifest files for CodeGen(2f9397)
  • Add Kubernetes manifest files for deploying CodeTrans(c9548d)
  • Updated READMEs for kubernetes example pipelines(c37d9c)
  • Update all examples yaml files of GMC in GenAIExample(290a74)
  • Doc: fix minor issue in GMC doc(d99461)
  • README for installing 4 worklods using helm chart(6e797f)
  • Update Kubernetes manifest files for deploying ChatQnA(665c46)
  • Add new example of SearchQnA for GenAIExample(21b7d1)
  • Add new example of Translation for GenAIExample(d0b028)

• Other examples

  • Update reranking microservice dockerfile path (d7a5b7)
  • Update tgi-gaudi version(3505bd)
  • Refine README of Examples(f73267)
  • Update READMEs(8ad7f3)
  • [CodeGen] Add codegen flowchart(377dd2)
  • Update audioqna image name(615f0d)
  • Add auto-truncate to gaudi tei (8d4209)
  • Update visualQnA chinese version(497895)
  • Fix Typo for Translation Example(95c13d)
  • FAQGen Megaservice(8c4a25)
  • Code-gen-react-ui(1b48e5)
  • Added doc sum react-ui(edf0d1)

• CI/UT

  • Frontend failed with unknown timeout issue (7ebe78)
  • Adding Chatqna Benchmark Test(11a56e)
  • Expand tgi connect timeout(ee0dcb)
  • Optimize gmc manifest e2e tests(15fc6f)
  • Add docker compose yaml print for test(bb4230)
  • Refactor translation ci test (b7975e)
  • Refactor searchqna ci test(ecf333)
  • Translate UT for UI(284d85)
  • Enhancement the codetrans e2e test(450efc)
  • Allow gmc e2e workflow to get secrets(f45f50)
  • Add checkout ref in gmc e2e workflow(62ae64)
  • SearchQnA UT(268d58)

GenAIComps

• Cores

  • Support https for microservice(2d6772)
  • Enlarge megaservice request timeout for supporting high concurrency(876ca5)
  • Add dynamic DAG(f2995a)

• LLM

  • Optional vllm microservice container build(963755)
  • Refine vllm instruction(6e2c28)
  • Introduce 'entrypoint.sh' for some Containers(9ecc5c)
  • Support llamaindex for retrieval microservice and remove langchain(61795f)
  • Update tgi with text-generation-inference:2.1.0(f23694)
  • Fix requirements(f4b029)
  • Add vLLM on Ray microservice(ec3b2e)
  • Update code/readme/UT for Ray Serve and VLLM([dd939c](https://gith...

Generative AI Examples v0.7 Release Notes

OPEA Highlight

• Add 3 MegaService examples: Translation, SearchQnA and AudioQnA
• Add 4 MicroService and LLM supports llamaIndex, vllm, RayServe
• Enable Dataprep: extract info from table, image...etc
• Add HelmChart and GenAI Microservice Connector(GMC) test

GenAIExamples

• ChatQnA

  • ChatQnA supports Qwen2(422b4b)
  • Add no_proxy in docker compose yaml for micro services(99eb6a, 240587)
  • Fix DataPrep image build in ChatQnA(2fb070)
  • Add Nvidia GPU support for ChatQnA(e80e56)
  • Update ChatQnA docker_compose.yaml to fix downloads failing(e948a7, f2a943)
  • Chat QNA React UI with conversation history(b994bc)
  • Adapt Chinese characters(2f4723)

• Other examples

  • Refactor Translation Example(409c723)
  • Add AudioQnA with GenAIComps(b4d8e1)
  • Add SearchQnA with GenAIComps(6b76a9)
  • Add env for searchqna(d9b62a)
  • Supports ASR on HPU(2a4860)
  • Fix DocSum Gaudi building instructions(29de55)
  • Add image build job in docker compose e2e gaudi test in CI(4fecd4)

• CI

  • Add docker build job in manifest e2e workflow(c5f309)
  • Create reuse workflow for get-test-matrix in CI(961abb)
  • Enable new CI runner and improve manifest e2e test scripts(26d6ea)
  • Enable building latest megaservice image on push event in CI(a0b94b)
  • Fix the image build refer(01eed8)
  • Add build docker image option for test scripts(e32a51)
  • Add e2e test of chatqna(afcb3a), codetrans(295b818), codegen(960cf38), docsum(2e62ecc))

GenAIComps

• Cores

  • Add aio orchestrator to boost concurrent serving(db3b4f)
  • Add microservice level perf statistics(597b3c, ba1d11)
  • Add Gateway for Translation(1b654d)

• LLM

  • Support Qwen2 in LLM Microservice(3f5cde)
  • Fix the vLLM docker compose issues(3d134d)
  • Enable vLLM Gaudi support for LLM service based on officially habana vllm release(0dedc2)
  • Openvino support in vllm(7dbad0)
  • Support Ollama microservice(a00e36)
  • Support vLLM XFT LLM microservice(2a6a29, 309c2d, fe5f39)
  • Add e2e test for llm summarization tgi(e8ebd9)

• DataPrep

  • Support Dataprep(f7443f), embedding(f37ce2) microservice with Llama Index
  • Fix dataprep microservice path issue(e20acc)
  • Add milvus microservice(e85033)
  • Add Ray version for multi file process(40c1aa)
  • Fix dataprep timeout issue(61ead4)
  • Add e2e test for dataprep redis langchain(6b7bec)
  • Supported image summarization with LVM in dataprep microservice(86412c)
  • Enable conditional splitting for html files(e1dad1)
  • Added support for pyspark in dataprep microservice(a5eb14)
  • DataPrep extract info from table in the docs(953e78)
  • Added support for extracting info from image in the docs(e23745)

• Other Components

  • Add PGvector support in Vectorstores(1b7001) and Retriever(75eff6), Dataprep(9de3c7)
  • Add Mosec embedding(f76685) and reranking(a58ca4)
  • Add knowledge graph components(4c0afd)
  • Add LVMs LLaVA component(bd385b)
  • Add asr/tts components for xeon and hpu(cef6ea)
  • Add WebSearch Retriever Microservice(900178)
  • Add initial pii detection microservice(e38041)
  • Pinecone support for dataprep and retrieval microservice(8b6486)
  • Support prometheus metrics for opea microservices(758914), (900178)
  • Add no_proxy env for micro services(df0c11)
  • Enable RAGAS(8a670e)
  • Fix RAG performance issues(70c23d)
  • Support rerank and retrieval of RAG OPT(b51675)
  • Reranking using an optimized bi-encoder(574847)
  • Use parameter for retriever(358dbd), reranker(dfdd08)

• CI

  • CI optimization to support multiple test for single kind of service(38f646)
  • Update CI to support dataprep_redis path level change(5c0773)
  • Enable python coverage(cd91cf)
  • Add codecov(da2689)
  • Enable microservice docker images auto build and push(16c5fd)

GenAIEvals

• Enable autorag to automatically generate the evaluation dataset and evaluate the RAG system(b24bff)
• Support document summar...

Generative AI Examples v0.6 Release Notes

OPEA Highlights

• Add 4 MegaService examples: CodeGen, ChatQnA, CodeTrans and Docsum, you can deploy them on Kubernetes
• Enable 10 microservices for LLM, RAG, security...etc
• Support text generation, code generation and end-to-end evaluation

GenAIExamples

• Build 4 reference solutions for some classic GenAI applications, like code generation, chat Q&A, code translation and document summarization, through orchestration interface in GenAIComps.
• Support seamlessly deployment on Intel Xeon and Gaudi platform through Kubernetes and Docker Compose.

GenAIComps

• Activate a suite of microservices including ASR, LLMS, Rerank, Embedding, Guardrails, TTS, Telemetry, DataPrep, Retrieval, and VectorDB. ASR functionality is fully operational on Xeon architecture, pending readiness on Gaudi. Retrieval capabilities are functional on LangChain, awaiting readiness on LlamaIndex. VectorDB functionality is supported on Redis, Chroma, and Qdrant, with readiness pending on SVS.
• Added 14 file formats support in data preparation microservices and enabled Safeguard of conversation in guardrails.
• Added the Ray Gaudi Supported for LLM Service.

GenAIEvals

• Add evaluating the models on text-generation tasks(lm-evaluation-harness) and coding tasks (bigcode-evaluation-harness)
• Add end-to-end evaluation with microservice

GenAIInfra

• Add Helm Charts redis-vector-db, TEI, TGI and CodeGen for deploying GenAIExamples on Kubernetes
• Add Manifests for deploying GenAIExamples CodeGen, ChatQnA and Docsum on Kubernetes and on Docker Compose

Generative AI Examples v0.1 Release Notes

• Highlights
• Examples

Highlights

• Provides a collective list of Generative AI (GenAI) and Retrieval-Augmented Generation (RAG) examples such as chatbot with question and answering (ChatQnA), code generation (CodeGen), document summary (DocSum), etc.
• All the examples are well-validated and optimized on Intel platforms.
• Use ecosystem-compliant APIs to build the end-to-end GenAI examples.
• Deploy the GenAI examples with performance on Intel platforms.

Examples

• ChatQnA: an example of chatbot for question and answering through retrieval argumented generation (RAG)
• CodeGen: an example of copilot designed for code generation in Visual Studio Code.
• DocSum: an example of chatbot for summarizing the content of documents or reports.
• SearchQnA: an example of chatbot for using search engine to enhance QA quality.
• VisualQnA is an example of chatbot for question and answering based on the images.

Validated Configurations

• Python 3.8, 3.9, 3.10, 3.11
• Ubuntu 20.04
• PyTorch 2.2.0+cpu 2.1.0+cpu
• Intel® Extension for PyTorch 2.2.0+cpu, 2.1.0+cpu
• TGI Gaudi 1.2.1
• LangChain 0.1.12