feat: add support for csv files #592

LMMilliken · 2022-10-31T16:50:11Z

This PR adds support for csv files to the finetuner.fit function.
Both train_data and eval_data can now be supplied as paths to a csv file or an in memory TextIO stream. The format of the provided csv can be any that are listed by the csv.list_dialects function. Currently, each row of the csv file can contain:

text-text pairs
image-image pairs
text-label pairs
image-label pairs
text-image pairs (for CLIP models)

Each row must of the same format, and to indicate that the second column represents labels and not text, provide a dictionary with is_labeled = True as the csv_options argument.

This PR references an open issue
I have added a line about this change to CHANGELOG

guenthermi

Added some comments

docs/get-started/how-it-works.md

docs/walkthrough/create-training-data.md

finetuner/utils.py

tests/unit/test_utils.py

CHANGELOG.md

docs/get-started/design-principles.md

docs/get-started/how-it-works.md

docs/walkthrough/create-training-data.md

finetuner/utils.py

Co-authored-by: George Mastrapas <32414777+gmastrapas@users.noreply.github.com> Co-authored-by: Michael Günther <guenthermi50@gmail.com>

bwanglzu

let's also add an integration test

docs/walkthrough/create-training-data.md

finetuner/experiment.py

finetuner/utils.py

bwanglzu

left some minor comments

finetuner/utils.py

gmastrapas

Nice! Some minor comments

README.md

docs/walkthrough/create-training-data.md

docs/walkthrough/run-job.md

docs/walkthrough/using-callbacks.md

finetuner/utils.py

Co-authored-by: George Mastrapas <32414777+gmastrapas@users.noreply.github.com>

bwanglzu

great job!

gmastrapas

LGTM!

tests/unit/test_data.py

guenthermi

LGTM, only added some minor comments

guenthermi · 2022-11-03T16:49:18Z

docs/walkthrough/create-training-data.md

+
+At the model saving time, you will discover, we are saving two models to your local directory. 
+```
+


I would suggest to add a note that csv data is before the finetuning loaded into memory (a DocumentArray object) and thereby locally stored images are also loaded into memory

guenthermi · 2022-11-03T17:06:15Z

finetuner/data.py

+def load_finetune_data_from_csv(
+    file: Union[str, TextIO],
+    task: str = 'text-to-text',
+    options: CSVOptions = CSVOptions(),


I think it is not recommended to call a constructor in a function declaration, since it is then only called once and every function while use the exact same instance.
So it is better to do:

def load_finetune_data_from_csv( [...] options=None, [...] ): options = options or CSVOptions()

github-actions · 2022-11-04T08:08:28Z

📝 Docs are deployed on https://ft-feat-support-csv--jina-docs.netlify.app 🎉

feat: add support for csv files

595b707

LMMilliken self-assigned this Oct 31, 2022

github-actions bot added size/s area/core area/entrypoint size/m and removed size/s labels Oct 31, 2022

LMMilliken force-pushed the feat-support-csv branch from 6704251 to 9f49717 Compare November 1, 2022 09:04

feat: add from_csv function

31199ff

LMMilliken force-pushed the feat-support-csv branch from 9f49717 to 31199ff Compare November 1, 2022 09:09

test: added tests

f2a3099

github-actions bot added the area/testing This issue/PR affects testing label Nov 1, 2022

LMMilliken linked an issue Nov 1, 2022 that may be closed by this pull request

Add support for csv files #593

Closed

lmmilliken added 2 commits November 1, 2022 16:40

fix: fix for test_create_run

93dde31

docs: started updating docs

9bf52e6

github-actions bot added the area/docs label Nov 2, 2022

LMMilliken force-pushed the feat-support-csv branch from d34e894 to 909247b Compare November 2, 2022 14:05

LMMilliken marked this pull request as ready for review November 2, 2022 14:09

LMMilliken requested review from gmastrapas, bwanglzu and guenthermi November 2, 2022 14:09

docs: updated documentation

623a446

LMMilliken force-pushed the feat-support-csv branch from 909247b to 623a446 Compare November 2, 2022 14:22

guenthermi reviewed Nov 2, 2022

View reviewed changes

gmastrapas requested changes Nov 2, 2022

View reviewed changes

Apply suggestions from code review

85d8366

Co-authored-by: George Mastrapas <32414777+gmastrapas@users.noreply.github.com> Co-authored-by: Michael Günther <guenthermi50@gmail.com>

bwanglzu requested changes Nov 3, 2022

View reviewed changes

docs/walkthrough/create-training-data.md Show resolved Hide resolved

finetuner/experiment.py Outdated Show resolved Hide resolved

finetuner/utils.py Outdated Show resolved Hide resolved

lmmilliken added 3 commits November 3, 2022 11:34

feat: added class

aec0e1b

fix: fix failing tests

0d8f0f8

chore: restructured method

e0f2203

LMMilliken requested review from guenthermi, bwanglzu and gmastrapas November 3, 2022 12:22

chore: renamed CSV_options to CSVOptions

be1ff52

bwanglzu requested changes Nov 3, 2022

View reviewed changes

finetuner/utils.py Outdated Show resolved Hide resolved

finetuner/utils.py Outdated Show resolved Hide resolved

finetuner/utils.py Outdated Show resolved Hide resolved

chore: corrected error message

f949e1e

gmastrapas requested changes Nov 3, 2022

View reviewed changes

LMMilliken and others added 2 commits November 3, 2022 14:12

Apply suggestions from code review

2864537

Co-authored-by: George Mastrapas <32414777+gmastrapas@users.noreply.github.com>

chore: renamed utils.py to data.py

5dec4b6

bwanglzu approved these changes Nov 3, 2022

View reviewed changes

test: added tests

7a3076f

github-actions bot added size/l and removed size/m labels Nov 3, 2022

LMMilliken requested a review from gmastrapas November 3, 2022 14:42

gmastrapas approved these changes Nov 3, 2022

View reviewed changes

tests/unit/test_data.py Outdated Show resolved Hide resolved

tests/unit/test_data.py Outdated Show resolved Hide resolved

LMMilliken force-pushed the feat-support-csv branch from dcb8084 to 17d199b Compare November 3, 2022 16:18

guenthermi approved these changes Nov 3, 2022

View reviewed changes

docs: updated readme

8cb3f77

LMMilliken force-pushed the feat-support-csv branch from 17d199b to 8cb3f77 Compare November 4, 2022 08:01

LMMilliken merged commit a3b62a0 into main Nov 4, 2022

LMMilliken deleted the feat-support-csv branch November 4, 2022 08:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for csv files #592

feat: add support for csv files #592

LMMilliken commented Oct 31, 2022 •

edited

Loading

guenthermi left a comment

bwanglzu left a comment

bwanglzu left a comment

gmastrapas left a comment

bwanglzu left a comment

gmastrapas left a comment

guenthermi left a comment

guenthermi Nov 3, 2022

guenthermi Nov 3, 2022

github-actions bot commented Nov 4, 2022


		At the model saving time, you will discover, we are saving two models to your local directory.
		```

feat: add support for csv files #592

feat: add support for csv files #592

Conversation

LMMilliken commented Oct 31, 2022 • edited Loading

guenthermi left a comment

Choose a reason for hiding this comment

bwanglzu left a comment

Choose a reason for hiding this comment

bwanglzu left a comment

Choose a reason for hiding this comment

gmastrapas left a comment

Choose a reason for hiding this comment

bwanglzu left a comment

Choose a reason for hiding this comment

gmastrapas left a comment

Choose a reason for hiding this comment

guenthermi left a comment

Choose a reason for hiding this comment

guenthermi Nov 3, 2022

Choose a reason for hiding this comment

guenthermi Nov 3, 2022

Choose a reason for hiding this comment

github-actions bot commented Nov 4, 2022

LMMilliken commented Oct 31, 2022 •

edited

Loading