Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add TranslationTask #396

Merged
merged 55 commits into from
Dec 11, 2023
Merged

Add TranslationTask #396

merged 55 commits into from
Dec 11, 2023

Conversation

rmitsch
Copy link
Collaborator

@rmitsch rmitsch commented Dec 5, 2023 •
edited
Loading

Description

Add TranslationTask.

Corresponding documentation PR

explosion/spaCy#13183

Types of change

New feature.

Checklist

  • I confirm that I have the right to submit this contribution under the project's MIT license.
  • I ran all tests in tests and usage_examples/tests, and all new and existing tests passed. This includes
    • all external tests (i. e. pytest ran with --external)
    • all tests requiring a GPU (i. e. pytest ran with --gpu)
  • My changes don't require a change to the documentation, or if they do, I've added all required information.

@rmitsch
Add context length info. Refactor BuiltinTask and models to facilitat…
3aba660 
…e this.
@rmitsch
Merge branch 'develop' into feat/inf-doc-len
5699773 
# Conflicts:
#	spacy_llm/tasks/builtin_task.py
@rmitsch
Add token count estimator plumbing.
4213372
@rmitsch
Add plumbing for mapper and reducer.
f440ca4
@rmitsch
Add ShardMapper prototype.
e47f762
@rmitsch
Integrating mapping into prompt generation workflow.
89a5510
@rmitsch
Update response parsing and component to support sharding (WIP).
086dec9
@rmitsch
Fix shard & prompt flow.
23718fc
@rmitsch
Fix shard & prompt flow.
7ce670d
@rmitsch
Remove todo comments.
0d75ea8
@rmitsch
Fix Anthropic, Cohere, NoOp model tests.
9da7098
@rmitsch
Merge branch 'develop' into feat/inf-doc-len
0cb9afd
@rmitsch
Fix test_llm_pipe().
f368412
@rmitsch
Fix type checking test.
b1f111d
@rmitsch
Fix span parsing tests.
44a2787
@rmitsch
Fix internal tests.
6d8cdc7
@rmitsch
Fix _CountTask.
e712f41
@rmitsch
Fix sentiment and summarization tasks and tests.
985fd68
@rmitsch
Fix Azure connection URL. Fix Model test pings.
98842a2
@rmitsch
Fix Lemma parsing.
b54a3d9
@rmitsch
Start work on doc-to-shard property copying.
9bf365d
@rmitsch
Fix REL doc preprocessing.
dddfaab
@rmitsch
Remove comment on doc attribute handling during sharding, as this is …
3af21b5 
…done by spaCy's slicing directly.
@rmitsch
Add reducer implementations.
fee9ca7
@rmitsch
Implement outstanding task reducers.
e508499
@rmitsch
Resolve merge conflicts.
3218541
@rmitsch
Add shardable/non-shardable LLM task typing distinction. Add support …
c104387 
…for handling both types of tasks. Update tests.
@rmitsch
Merge branch 'develop' into feat/inf-doc-len
2c6d899 
# Conflicts:
#	spacy_llm/tests/tasks/test_rel.py
@rmitsch
Fix EL task.
2502c4d
@rmitsch
Fix EL tokenization and highlighting partially.
03055c5
@rmitsch rmitsch added feat/new New feature feat/task Feature: tasks labels Dec 5, 2023
@rmitsch rmitsch self-assigned this Dec 5, 2023
@rmitsch rmitsch changed the base branch from main to feat/inf-doc-len December 5, 2023 14:47
@rmitsch rmitsch mentioned this pull request Dec 5, 2023
Merged
3 tasks
@rmitsch rmitsch marked this pull request as ready for review December 5, 2023 15:08
@rmitsch rmitsch added the Test external Run external tests label Dec 5, 2023
svlandeg
svlandeg approved these changes Dec 7, 2023
View reviewed changes
Copy link
Member

@svlandeg svlandeg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Looking good. I suggest we merge this after the infinite-docs one, so this one will show up by itself in the commit history.

Base automatically changed from feat/inf-doc-len to develop December 11, 2023 14:27
@rmitsch
Merge branch 'develop' into feat/translation-task
9ffa746 
# Conflicts:
#	spacy_llm/models/hf/base.py
#	spacy_llm/models/hf/falcon.py
#	spacy_llm/models/hf/llama2.py
#	spacy_llm/models/hf/mistral.py
#	spacy_llm/models/hf/openllama.py
#	spacy_llm/models/hf/stablelm.py
#	spacy_llm/models/langchain/model.py
#	spacy_llm/pipeline/llm.py
#	spacy_llm/tasks/entity_linker/util.py
#	spacy_llm/tests/models/test_dolly.py
#	spacy_llm/tests/models/test_langchain.py
#	spacy_llm/tests/pipeline/test_llm.py
#	spacy_llm/tests/sharding/test_sharding.py
#	spacy_llm/tests/tasks/legacy/test_ner.py
#	spacy_llm/tests/tasks/legacy/test_spancat.py
#	spacy_llm/tests/tasks/test_textcat.py
#	spacy_llm/tests/test_combinations.py
#	spacy_llm/ty.py
#	usage_examples/tests/test_readme_examples.py
@rmitsch
Resolve merge conflicts.
c840fda
@rmitsch
Merge branch 'develop' into feat/translation-task
2c691b3 
# Conflicts:
#	spacy_llm/tests/sharding/test_sharding.py
@rmitsch
Fix test_llm_task_factories.
e3f4d73
@rmitsch rmitsch merged commit f628afc into develop Dec 11, 2023
10 checks passed
@svlandeg svlandeg deleted the feat/translation-task branch December 11, 2023 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@svlandeg svlandeg svlandeg approved these changes

Assignees

@rmitsch rmitsch

Labels
feat/new New feature feat/task Feature: tasks Test external Run external tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@rmitsch @svlandeg