[Bug]Data-prep failed on ingesting files

[chensuyue]

Priority

P1-Stopper

OS type

Ubuntu

Hardware type

GPU-AMD

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source
• Other

Deploy method

• Docker
• Docker Compose
• Kubernetes Helm Charts
• Kubernetes GMC
• Other

Running nodes

Single Node

What's the version?

Main branch, 2204fe8

Description

Test failed in several CI,
https://github.com/opea-project/GenAIExamples/actions/runs/13948828466/job/39042658138?pr=1693
https://github.com/opea-project/GenAIExamples/actions/runs/13940007456/job/39019740990?pr=1654

Reproduce steps

Run ChatQnA/tests/test_compose_on_rocm.sh

Raw log

[ tei-embedding ] Content is as expected.
+ sleep 1m
+ echo 'Deep learning is a subset of machine learning that utilizes neural networks with multiple layers to analyze various levels of abstract data representations. It enables computers to identify patterns and make decisions with minimal human intervention by learning from large amounts of data.'
+ validate_service http://10.53.22.29:6007/v1/dataprep/ingest 'Data preparation succeeded' dataprep_upload_file dataprep-redis-server
+ local URL=http://10.53.22.29:6007/v1/dataprep/ingest
+ local 'EXPECTED_RESULT=Data preparation succeeded'
+ local SERVICE_NAME=dataprep_upload_file
+ local DOCKER_NAME=dataprep-redis-server
+ local INPUT_DATA=
+ [[ dataprep_upload_file == *\d\a\t\a\p\r\e\p\_\u\p\l\o\a\d\_\f\i\l\e* ]]
+ cd /home/huggingface/OPEA-CICD/actions-runner/_work/GenAIExamples/GenAIExamples/ChatQnA/tests
++ curl --silent --write-out 'HTTPSTATUS:%{http_code}' -X POST -F files=@./dataprep_file.txt -H 'Content-Type: multipart/form-data' http://10.53.22.29:6007/v1/dataprep/ingest
+ HTTP_RESPONSE=HTTPSTATUS:000
Error: Process completed with exit code 7.

Attachments

No response

[xiguiw]

• echo '[ tei-embedding ] Content is as expected.'
• sleep 1s
  [ tei-embedding ] Content is as expected.
• sleep 1m
• echo 'Deep learning is a subset of machine learning that utilizes neural networks with multiple layers to analyze various levels > of abstract data representations. It enables computers to identify patterns and make decisions with minimal human intervention > by learning from large amounts of data.'
• validate_service http://10.53.22.29:6007/v1/dataprep/ingest 'Data preparation succeeded' dataprep_upload_file dataprep-> > > redis-server
• local URL=http://10.53.22.29:6007/v1/dataprep/ingest
• local 'EXPECTED_RESULT=Data preparation succeeded'
• local SERVICE_NAME=dataprep_upload_file
• local DOCKER_NAME=dataprep-redis-server
• local INPUT_DATA=
• [[ dataprep_upload_file == \d\a\t\a\p\r\e\p_\u\p\l\o\a\d_\f\i\l\e ]]
• cd /home/huggingface/OPEA-CICD/actions-runner/_work/GenAIExamples/GenAIExamples/ChatQnA/tests
  ++ curl --silent --write-out 'HTTPSTATUS:%{http_code}' -X POST -F files=@./dataprep_file.txt -H 'Content-Type: multipart/form-> data' http://10.53.22.29:6007/v1/dataprep/ingest
• HTTP_RESPONSE=HTTPSTATUS:000

I can reproduce this issue on chatQnA Xeon qdrant locally.

It looks like data-prep issue.
This happened random.
The data-prep are not ready sometime to accept the input request.

curl --silent --write-out 'HTTPSTATUS:%{http_code}' -X POST -F files=@./dataprep_file.txt -H 'Content-Type: multipart/form-data' http://10.53.22.29:6007/v1/dataprep/ingest

2025-03-20T09:35:16.378851342Z /usr/local/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
2025-03-20T09:35:16.378997022Z   warnings.warn(
2025-03-20T09:35:16.379025582Z /usr/local/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
2025-03-20T09:35:16.379056142Z   warnings.warn(
2025-03-20T09:35:52.100652942Z [2025-03-20 09:35:52,099] [    INFO] - opea_dataprep_qdrant - title='qdrant - vector search engine' version='1.13.4' commit='7abc684361fb81d8b62cf1554d8bf4fb65a706d7'
2025-03-20T09:35:52.107381462Z [2025-03-20 09:35:52,106] [    INFO] - Base service - CORS is enabled.
2025-03-20T09:35:52.109219982Z [2025-03-20 09:35:52,108] [    INFO] - Base service - Setting up HTTP server
2025-03-20T09:35:52.111509862Z [2025-03-20 09:35:52,110] [    INFO] - Base service - Uvicorn server setup on port 5000
2025-03-20T09:35:52.118304222Z INFO:     Waiting for application startup.
2025-03-20T09:35:52.119563022Z INFO:     Application startup complete.
2025-03-20T09:35:52.120206142Z INFO:     Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
2025-03-20T09:35:52.120285102Z [2025-03-20 09:35:52,119] [    INFO] - Base service - HTTP server setup successful
2025-03-20T09:35:52.128500862Z [2025-03-20 09:35:52,127] [    INFO] - opea_dataprep_microservice - OPEA Dataprep Microservice is starting...

The output status of above command is 0, no log in data-prep.

if you resend the command, it got success. data-prep log show accepts input request and status OK.

[letonghan]

I will check this issue locally.

[xiguiw]

REMIND: It's not a hardware specific issue.

[xiguiw]

@letonghan

data-prep comps test pass at the condition it starts ingesting file after waiting 1 minutes

+ cd /home/xiguiwan/OPEA/GenAIComps/comps/dataprep/deployment/docker_compose/
+ docker compose up redis-vector-db tei-embedding-serving dataprep-redis -d

https://github.com/opea-project/GenAIComps/blob/411af51b0bffb15ece82d23f782a8f960748e688/tests/dataprep/test_dataprep_redis.sh#L44

If sleep 10s after stat up then ingest files, the test failed, the log is the same as that in the GenAIExamples test log.
It seemed data-prep is not start up completly.

[+] Running 3/3
 ✔ Container redis-vector-db        Healthy                                                                                                          8.9s
 ✔ Container tei-embedding-serving  Healthy                                                                                                         23.9s
 ✔ Container dataprep-redis-server  Started                                                                                                         24.9s
+ sleep 10s
+ validate_microservice
+ delete_all 10.112.238.202 11108
+ local fqdn=10.112.238.202
+ local port=11108
+ shift 2
+ _invoke_curl 10.112.238.202 11108 delete -d '{"file_path":"all"}'
+ local url=http://10.112.238.202:11108/v1/dataprep/delete
+ local action=delete
+ shift 3
+ local header=
+ case $action in
+ header='Content-Type: application/json'
++ curl --silent --write-out 'HTTPSTATUS:%{http_code}' -X POST -H 'Content-Type: application/json' http://10.112.238.202:11108/v1/dataprep/delete -d '{"file_path":"all"}'
+ HTTP_RESPONSE=HTTPSTATUS:000
++ echo HTTPSTATUS:000
++ tr -d '\n'
++ sed -e 's/.*HTTPSTATUS://'
+ HTTP_STATUS=000
++ echo HTTPSTATUS:000
++ sed -e 's/HTTPSTATUS\:.*//g'
+ RESPONSE_BODY=
+ check_result 'dataprep - del' '{"status":true}' dataprep-redis-server /home/xiguiwan/OPEA/GenAIComps/tests/dataprep_del.log
+ local 'service_name=dataprep - del'
+ local 'expected_response={"status":true}'
+ local container_name=dataprep-redis-server
+ local logfile=/home/xiguiwan/OPEA/GenAIComps/tests/dataprep_del.log
+ local http_status=200
+ '[' 000 -ne 200 ']'
+ echo '[ dataprep - del ] HTTP status is not 200. Received status was 000'
[ dataprep - del ] HTTP status is not 200. Received status was 000
+ docker logs dataprep-redis-server
/usr/local/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
  warnings.warn(
/usr/local/lib/python3.11/site-packages/langchain/__init__.py:30: UserWarning: Importing PromptTemplate from langchain root module is no longer supported. Please use langchain_core.prompts.PromptTemplate instead.
  warnings.warn(
+ exit 1