Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support markdown to text function in text operator #149

Merged
merged 8 commits into from
Jun 4, 2024
Merged

feat: support markdown to text function in text operator #149

merged 8 commits into from
Jun 4, 2024

Conversation

chuang8511
Copy link
Member

Because

  • we need to support markdown file for knowledge base

This commit

  • convert markdown file to text in text operator

@linear Linear
Copy link

linear bot commented May 31, 2024

INS-4797 [Component][Text Operator] convert md file to text function in TASK_CONVERT

@chuang8511 chuang8511 marked this pull request as ready for review May 31, 2024 11:59
donch1989
donch1989 requested changes May 31, 2024
View reviewed changes
operator/text/v0/convert.go Outdated Show resolved Hide resolved
@chuang8511 chuang8511 requested a review from donch1989 June 3, 2024 10:55
@chuang8511
Copy link
Member Author

Quite not sure why rerunning the codecov will solve the problem of failing unit test of document component.
Will dig in and fix it later.

@donch1989
Copy link
Member

donch1989 commented Jun 3, 2024 •
edited
Loading

Quite not sure why rerunning the codecov will solve the problem of failing unit test of document component. Will dig in and fix it later.

Try to move

component/operator/document/v0/convert_pdf_to_markdown.go

Line 39 in 365d49a

defer stdin.Close()
out of goroutine to L35? It might be caused by some concurrency problem since you defer the writer in goroutine, then it become flaky.

@chuang8511 chuang8511 requested a review from donch1989 June 3, 2024 14:05
@donch1989 donch1989 merged commit dcbae37 into main Jun 4, 2024
8 checks passed
@donch1989 donch1989 deleted the chunhao/ins-4797 branch June 4, 2024 17:01
@droplet-bot droplet-bot mentioned this pull request Jun 3, 2024
Merged
donch1989 pushed a commit that referenced this pull request Jun 6, 2024
@droplet-bot
chore(main): release 0.19.0-beta (#141)
40363c3 
🤖 I have created a release *beep* *boop*
---


##
[0.19.0-beta](v0.18.0-beta...v0.19.0-beta)
(2024-06-05)


### Features

* add pdf component
([#138](#138))
([517afcf](517afcf))
* add task chunk text
([#139](#139))
([7b36553](7b36553))
* **instill:** adopt latest Model endpoints
([#146](#146))
([7f2537b](7f2537b))
* optimise ux for slack component
([#143](#143))
([ed60235](ed60235))
* refactor package structure
([#140](#140))
([4853d4c](4853d4c))
* support markdown to text function in text operator
([#149](#149))
([dcbae37](dcbae37))
* unify component interface
([#144](#144))
([ad35e10](ad35e10))


### Bug Fixes

* bug of failure of document component
([#152](#152))
([aed51f8](aed51f8))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
namwoam pushed a commit to namwoam/component that referenced this pull request Jun 24, 2024
@chuang8511
feat: support markdown to text function in text operator (instill-ai#149
9ba01a4 
)

Because

- we need to support markdown file for knowledge base

This commit

- convert markdown file to text in text operator
namwoam pushed a commit to namwoam/component that referenced this pull request Jun 24, 2024
@droplet-bot
chore(main): release 0.19.0-beta (instill-ai#141)
8bc32ee 
🤖 I have created a release *beep* *boop*
---


##
[0.19.0-beta](instill-ai/component@v0.18.0-beta...v0.19.0-beta)
(2024-06-05)


### Features

* add pdf component
([instill-ai#138](instill-ai#138))
([517afcf](instill-ai@517afcf))
* add task chunk text
([instill-ai#139](instill-ai#139))
([7b36553](instill-ai@7b36553))
* **instill:** adopt latest Model endpoints
([instill-ai#146](instill-ai#146))
([7f2537b](instill-ai@7f2537b))
* optimise ux for slack component
([instill-ai#143](instill-ai#143))
([ed60235](instill-ai@ed60235))
* refactor package structure
([instill-ai#140](instill-ai#140))
([4853d4c](instill-ai@4853d4c))
* support markdown to text function in text operator
([instill-ai#149](instill-ai#149))
([dcbae37](instill-ai@dcbae37))
* unify component interface
([instill-ai#144](instill-ai#144))
([ad35e10](instill-ai@ad35e10))


### Bug Fixes

* bug of failure of document component
([instill-ai#152](instill-ai#152))
([aed51f8](instill-ai@aed51f8))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@donch1989 donch1989 donch1989 approved these changes

@pinglin pinglin Awaiting requested review from pinglin pinglin is a code owner

@xiaofei-du xiaofei-du Awaiting requested review from xiaofei-du xiaofei-du is a code owner

@jvallesm jvallesm Awaiting requested review from jvallesm jvallesm is a code owner

Assignees
No one assigned
Labels
instill component
Projects
🔮 Instill Core
Status: 👋 Done
Milestone
No milestone
3 participants
@chuang8511 @donch1989 @droplet-bot