Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add dependency for pdf2image and OCR processing #421

Merged
merged 9 commits into from
Aug 21, 2024
Merged

Add dependency for pdf2image and OCR processing #421

merged 9 commits into from
Aug 21, 2024

Conversation

ZailiWang
Copy link
Collaborator

Description

Fix #414 as well as adding another tesseract-ocr dependency.

Issues

#414

Type of change

List the type of change like below. Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds new functionality)
  • Breaking change (fix or feature that would break existing design and interface)
  • Others (enhancement, documentation, validation, etc.)

Dependencies

poppler-utils
tesseract-ocr

Tests

Tested locally with langchain docker image.

@ZailiWang
Copy link
Collaborator Author

Testing with the file used in #414, the error is fixed.

Progress: |██████████████████████████████████████████████████| 100.0% CompleteDone preprocessing. Created  343  chunks of the original pdf
[ ingest chunks ] file name: nke-10k-2023.pdf
[ ingest chunks ] Current batch: 0

@ZailiWang
Copy link
Collaborator Author

@lvliang-intel would you kindly help review the PR? Thanks!

@ZailiWang ZailiWang requested a review from lvliang-intel August 7, 2024 06:45
@lvliang-intel
Copy link
Collaborator

@ZailiWang,
Please also add the dependency libs into the BOM.

@ZailiWang
Copy link
Collaborator Author

Would you tell where's the BOM file of this repo? I didn't find it.

lkk12014402 pushed a commit that referenced this pull request Aug 8, 2024
@WenjiaoYue
SearchQnA UT (#421)
268d58d 
Signed-off-by: Yue, Wenjiao <wenjiao.yue@intel.com>
@chensuyue
Copy link
Collaborator

Would you tell where's the BOM file of this repo? I didn't find it.

Please ping me by teams, I will send you the link.

@chensuyue chensuyue merged commit 9397522 into opea-project:main Aug 21, 2024
11 checks passed
chensuyue pushed a commit that referenced this pull request Aug 21, 2024
@ZailiWang @chensuyue
Add dependency for pdf2image and OCR processing (#421)
f481f44 
(cherry picked from commit 9397522)
tileintel pushed a commit to siddhivelankar23/GenAIComps that referenced this pull request Aug 22, 2024
@ZailiWang @siddhivelankar23
Add dependency for pdf2image and OCR processing (opea-project#421)
7bb4cdf 
Signed-off-by: siddhivelankar23 <siddhi.velankar@intel.com>
srinarayan-srikanthan pushed a commit to siddhivelankar23/GenAIComps that referenced this pull request Sep 1, 2024
@ZailiWang @srinarayan-srikanthan
Add dependency for pdf2image and OCR processing (opea-project#421)
7e660a1 
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>
sharanshirodkar7 pushed a commit to predictionguard/pg-GenAIComps that referenced this pull request Sep 3, 2024
@ZailiWang @sharanshirodkar7
Add dependency for pdf2image and OCR processing (opea-project#421)
30aaedd
yinghu5 pushed a commit that referenced this pull request Sep 4, 2024
@srinarayan-srikanthan @pre-commit-ci
@XuehaoSun @letonghan @ZailiWang @xuechendi @Spycsh @BaoHuiling @XuhuiRen @XinyuYe-Intel @lkk12014402 @chensuyue @ashahba @qgao007
adding embedding support for CLIP based models for VideoRAGQnA exampl…
2a53e25 
…e for v0.9 (#538)

* clip embedding support

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* test script for embedding

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* fix freeze workflow (#522)

Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Fix Dataprep Potential Error in get_file (#540)

* fix get file error & refine logs

Signed-off-by: letonghan <letong.han@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <letong.han@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Support SearchedDoc input type in LLM for No Rerank Pipeline (#541)

Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add dependency for pdf2image and OCR processing (#421)

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add local_embedding return 768 length to align with chatqna example (#313)

Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* add telemetry doc (#536)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add video-llama LVM microservice under lvms  (#495)

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Fix the data load issue for structured files (#505)

Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add finetuning component (#502)

Signed-off-by: Xinyu Ye <xinyu.ye@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
Co-authored-by: test <test@intel.com>
Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* add torchvision into requirements (#546)

Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Use Gaudi base images from Dockerhub (#526)

* Use Gaudi base images from Dockerhub

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Fixing the malformed tag

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* fix another malformed tag

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

---------

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Add toxicity detection microservice (#338)

* Add toxicity detection microservice

Signed-off-by: Qun Gao <qun.gao@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Modification to toxicity plugin PR  (#432)

* changed microservice to use Service.GUARDRAILS and input/output to TextDoc

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>

* simplify dockerfile to use langchain

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>

* sort requirements

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Minor SPDX header update (#434)

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Remove 'langsmith' per code review (#534)

Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>

* Add toxicity detection microservices with E2E testing

Signed-off-by: Qun Gao <qun.gao@intel.com>

---------

Signed-off-by: Qun Gao <qun.gao@intel.com>
Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Co-authored-by: Tyler W <tyler.wilbers@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* rename script and use 5xxx

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* add proxy for build

Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* fixed commit issues

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* Fix docarray constraint

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* updated docarray

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rm telemetry which cause error in mega

Signed-off-by: BaoHuiling <huiling.bao@intel.com>

* renamed dirs

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

* renamed test

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>

---------

Signed-off-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>
Signed-off-by: Sun, Xuehao <xuehao.sun@intel.com>
Signed-off-by: letonghan <letong.han@intel.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: BaoHuiling <huiling.bao@intel.com>
Signed-off-by: XuhuiRen <xuhui.ren@intel.com>
Signed-off-by: Xinyu Ye <xinyu.ye@intel.com>
Signed-off-by: chensuyue <suyue.chen@intel.com>
Signed-off-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Signed-off-by: Qun Gao <qun.gao@intel.com>
Signed-off-by: Tyler Wilbers <tyler.wilbers@intel.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sun, Xuehao <xuehao.sun@intel.com>
Co-authored-by: Letong Han <106566639+letonghan@users.noreply.github.com>
Co-authored-by: Zaili Wang <109502517+ZailiWang@users.noreply.github.com>
Co-authored-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: Sihan Chen <39623753+Spycsh@users.noreply.github.com>
Co-authored-by: Huiling Bao <huiling.bao@intel.com>
Co-authored-by: XuhuiRen <44249229+XuhuiRen@users.noreply.github.com>
Co-authored-by: XinyuYe-Intel <xinyu.ye@intel.com>
Co-authored-by: lkk <33276950+lkk12014402@users.noreply.github.com>
Co-authored-by: test <test@intel.com>
Co-authored-by: root <root@idc708073.jf.intel.com>
Co-authored-by: chen, suyue <suyue.chen@intel.com>
Co-authored-by: Abolfazl Shahbazi <abolfazl.shahbazi@intel.com>
Co-authored-by: qgao007 <108324932+qgao007@users.noreply.github.com>
Co-authored-by: Tyler W <tyler.wilbers@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chensuyue chensuyue chensuyue approved these changes

@lvliang-intel lvliang-intel lvliang-intel approved these changes

@XinyuYe-Intel XinyuYe-Intel Awaiting requested review from XinyuYe-Intel XinyuYe-Intel is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

dataprep-redis: failed when uploading pdf with process_table=true
3 participants
@ZailiWang @lvliang-intel @chensuyue