feat: add pdf component #138

chuang8511 · 2024-05-20T20:22:37Z

Because

We want users to transform PDF to markdown format file

This commit

Transform PDF to markdown in VDP.

Note

Python code should have its own test code. So, I mock cmd.
- However, the current pdfTransformer is PoC for the first phase, so I did not add the test code for python.

linear · 2024-05-20T20:22:39Z

INS-3098 [Component][PDF Operator] add TASK_CONVERT_TO_MARKDOWN

pkg/operator/pdf/v0/config/definition.json

pkg/operator/pdf/v0/config/tasks.json

pkg/operator/pdf/v0/convertPdfToMarkdown.go

pkg/operator/pdf/v0/main.go

pkg/operator/pdf/v0/convertPdfToMarkdown_test.go

pkg/operator/pdf/v0/python/pdfTransformer.py

chuang8511

I replied comments and will push the modification soon!

pkg/operator/pdf/v0/main.go

pkg/operator/pdf/v0/python/pdfTransformer.py

pkg/operator/pdf/v0/mockCommand.go

chuang8511 · 2024-05-22T13:57:31Z

I was trying to look for where we set this Pdf.
But, it seems it transforms in frontend.

pkg/mock/generator.go

jvallesm · 2024-05-23T07:18:46Z

I was trying to look for where we set this Pdf. But, it seems it transforms in frontend.

I see, I think it would be beneficial to have a title field in the base task object, so we only rely on the frontend transformation as a fallback. In cases where we have acronyms like here we can't just use the simplistic transformation (_ -> + Title case). Please report this to frontend / product so we can prioritise this. For now I don't think this should prevent us from shipping the feature.

jvallesm

Thanks for addressing my suggestions. There are a few unaddressed comments but they're design / cosmetic suggestions rather than blocking change requests. Let me know what you think about them, if you intend to address any of them, and let's move forward when all the conversations are resolved.

pkg/mock/generator.go

pkg/mock/usage_handler_mock.gen.go

pkg/operator/pdf/v0/main.go

pkg/operator/pdf/v0/struct.go

pkg/operator/pdf/v0/convert_pdf_to_markdown_test.go

pkg/operator/pdf/v0/convertPdfToMarkdown_test.go

pkg/operator/pdf/v0/main.go

pkg/operator/pdf/v0/python/pdfTransformer.py

chuang8511 · 2024-05-23T09:58:18Z

@jvallesm
#138 (comment)

I am not quite familiar with GitHub UXUI.
I have taken this out and fixed all Pdf to PDF already!
Just sync this information here with you.

pkg/operator/pdf/v0/convert_pdf_to_markdown_test.go

jvallesm

🏋️ 💦 Thanks for the thorough revision.

This is good to go 🚀

chuang8511 · 2024-05-23T20:31:05Z

For the reviewers @jvallesm (( cc'd @donch1989
I tested with Dockerfile, and it seems the code is compiled.
So, it seems I cannot execute the code with the way by the path.

So, I am testing to put all python code in Go file and execute it directly, which I also do not like.
But, it seems to be the easy way to deal with it.

I will push my code on Friday and notify you again to sync with you the result.

donch1989 · 2024-05-27T01:54:48Z

@chuang8511
You can try to use go embed. This can help you embed non-golang resources in the golang package.

donch1989 · 2024-05-27T10:56:29Z

Hi @chuang8511
We did some structural refactoring in this PR. You will need to rebase your branch. I guess it won't be too difficult since most of the changes are just moving files.

donch1989 · 2024-05-30T03:04:29Z

@chuang8511
I'll merge this PR first. We can fix any bugs later if there are any.

🤖 I have created a release *beep* *boop* --- ## [0.19.0-beta](v0.18.0-beta...v0.19.0-beta) (2024-06-05) ### Features * add pdf component ([#138](#138)) ([517afcf](517afcf)) * add task chunk text ([#139](#139)) ([7b36553](7b36553)) * **instill:** adopt latest Model endpoints ([#146](#146)) ([7f2537b](7f2537b)) * optimise ux for slack component ([#143](#143)) ([ed60235](ed60235)) * refactor package structure ([#140](#140)) ([4853d4c](4853d4c)) * support markdown to text function in text operator ([#149](#149)) ([dcbae37](dcbae37)) * unify component interface ([#144](#144)) ([ad35e10](ad35e10)) ### Bug Fixes * bug of failure of document component ([#152](#152)) ([aed51f8](aed51f8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Because - We want users to transform PDF to markdown format file This commit - Transform PDF to markdown in VDP. Note - Python code should have its own test code. So, I mock `cmd`. - However, the current pdfTransformer is PoC for the first phase, so I did not add the test code for python.

🤖 I have created a release *beep* *boop* --- ## [0.19.0-beta](instill-ai/component@v0.18.0-beta...v0.19.0-beta) (2024-06-05) ### Features * add pdf component ([instill-ai#138](instill-ai#138)) ([517afcf](instill-ai@517afcf)) * add task chunk text ([instill-ai#139](instill-ai#139)) ([7b36553](instill-ai@7b36553)) * **instill:** adopt latest Model endpoints ([instill-ai#146](instill-ai#146)) ([7f2537b](instill-ai@7f2537b)) * optimise ux for slack component ([instill-ai#143](instill-ai#143)) ([ed60235](instill-ai@ed60235)) * refactor package structure ([instill-ai#140](instill-ai#140)) ([4853d4c](instill-ai@4853d4c)) * support markdown to text function in text operator ([instill-ai#149](instill-ai#149)) ([dcbae37](instill-ai@dcbae37)) * unify component interface ([instill-ai#144](instill-ai#144)) ([ad35e10](instill-ai@ad35e10)) ### Bug Fixes * bug of failure of document component ([instill-ai#152](instill-ai#152)) ([aed51f8](instill-ai@aed51f8)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

chuang8511 added 2 commits May 20, 2024 13:13

feat: Add pdf operator structure

9a1c242

feat: Add pdf component

841b6a6

droplet-bot added the instill vdp label May 20, 2024

feat: Finish PDF component with the transformer for pdf to markdown

8b0bc7d

droplet-bot added the instill component label May 21, 2024

chuang8511 added 2 commits May 21, 2024 13:40

fix: Fix lint error

7c5246b

fix: Take out metadata first to avoid the bugs

620529e

chuang8511 marked this pull request as ready for review May 21, 2024 13:07

chuang8511 requested review from donch1989, pinglin, xiaofei-du and jvallesm as code owners May 21, 2024 13:07

fix: Fix table bug

e7e936a

jvallesm reviewed May 22, 2024

View reviewed changes

chuang8511 added 3 commits May 22, 2024 11:44

fix: Modify naming to make the code follow Go convention

9a73856

fix: Make the internal part unexported

03bbbc0

fix: Import minimock to check the input and mock the response

9a9e2f8

chuang8511 commented May 22, 2024

View reviewed changes

pkg/operator/pdf/v0/main.go Outdated Show resolved Hide resolved

pkg/operator/pdf/v0/main.go Outdated Show resolved Hide resolved

pkg/operator/pdf/v0/python/pdfTransformer.py Outdated Show resolved Hide resolved

pkg/operator/pdf/v0/mockCommand.go Outdated Show resolved Hide resolved

chuang8511 added 2 commits May 22, 2024 14:58

fix: Fix typo

7f33934

fix: Fix typo and take out the unnecessary comments

c0c3eff

chuang8511 commented May 22, 2024

View reviewed changes

pkg/mock/generator.go Outdated Show resolved Hide resolved

chuang8511 requested a review from jvallesm May 22, 2024 14:23

jvallesm reviewed May 23, 2024

View reviewed changes

fix: Take out the comment and fix minimock command

3f65952

jvallesm reviewed May 23, 2024

View reviewed changes

pkg/operator/pdf/v0/convert_pdf_to_markdown_test.go Outdated Show resolved Hide resolved

fix: Fix the minimock version and test code

13c4ee3

chuang8511 requested a review from jvallesm May 23, 2024 13:19

chore: Change the name of task

3d6d41d

jvallesm previously approved these changes May 23, 2024

View reviewed changes

fix: Remove internal comments

3cbe0ab

chuang8511 marked this pull request as draft May 23, 2024 17:56

fix: Compiled code cannot source the python code by path

c8273fe

chuang8511 dismissed jvallesm’s stale review via c8273fe May 24, 2024 07:50

chore: change name from PDF to Markdown

98b868b

chuang8511 added 4 commits May 28, 2024 09:16

chore: Add document icon

75d6d71

chore: Rebase the branch and fix conflict

c9aceb6

fix: Use embed package to source Python code

8108717

chore: Modify the directory path

eb6979a

chuang8511 marked this pull request as ready for review May 28, 2024 11:29

donch1989 approved these changes May 30, 2024

View reviewed changes

donch1989 merged commit 517afcf into main May 30, 2024
8 checks passed

donch1989 deleted the chunhao/ins-3098 branch May 30, 2024 03:04

droplet-bot mentioned this pull request May 27, 2024

chore(main): release 0.19.0-beta #141

Merged

chuang8511 mentioned this pull request Jun 18, 2024

feat: add Anthropic component #161

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add pdf component #138

feat: add pdf component #138

chuang8511 commented May 20, 2024 •

edited

Loading

linear bot commented May 20, 2024

chuang8511 left a comment

chuang8511 commented May 22, 2024

jvallesm commented May 23, 2024

jvallesm left a comment

chuang8511 commented May 23, 2024

jvallesm left a comment

chuang8511 commented May 23, 2024

donch1989 commented May 27, 2024 •

edited

Loading

donch1989 commented May 27, 2024

donch1989 commented May 30, 2024

feat: add pdf component #138

feat: add pdf component #138

Conversation

chuang8511 commented May 20, 2024 • edited Loading

linear bot commented May 20, 2024

chuang8511 left a comment

Choose a reason for hiding this comment

chuang8511 commented May 22, 2024

jvallesm commented May 23, 2024

jvallesm left a comment

Choose a reason for hiding this comment

chuang8511 commented May 23, 2024

jvallesm left a comment

Choose a reason for hiding this comment

chuang8511 commented May 23, 2024

donch1989 commented May 27, 2024 • edited Loading

donch1989 commented May 27, 2024

donch1989 commented May 30, 2024

chuang8511 commented May 20, 2024 •

edited

Loading

donch1989 commented May 27, 2024 •

edited

Loading