Improve data prep #122

arnocandel · 2023-05-08T16:25:25Z

WIP: Improve data prep by adding <human>: at end, hopefully reduces rambling.

Increase deberta grading threshold.

arnocandel added 6 commits May 8, 2023 08:48

Add Fortune2000 data prep.

acc3d63

WIP: Improve data prep by adding <human>: at end, hopefully reduces rambling.

Remove space after human's message, before bot. Add human at end.

d193fcd

Add link to wikitext data.

08f28cc

End with <human>:\n for human_bot

1083509

Undo chat_sep

f58ca86

Add \n before ending <human>:

5217ab0

arnocandel force-pushed the improve-data-prep branch 3 times, most recently from d84d423 to 1e58a72 Compare May 8, 2023 20:01

Refactor unhelpful list.

747e580

Increase deberta grading threshold.

arnocandel force-pushed the improve-data-prep branch from 1e58a72 to 747e580 Compare May 8, 2023 20:03

arnocandel added 11 commits May 8, 2023 13:09

Shorten OIG data. Add proper ending \n<human>:

a615630

Update file names.

799a565

Increase batch size.

9257e59

Add deberta graded oasst1 with new \n<human>: ending.

1bfc24b

Improve OIG data quality by keeping sentences for long data.

f7d2027

Fix case of short sentences.

bd457db

Fix comment.

e48027a

Use TrainingArguments, more general.

70f3be8

Re-use previously created file.

f0104c1

Update dataset used for stats.

997c50a

Add row_id for oig_oasst1 and oasst1 data.

70275a3

arnocandel force-pushed the improve-data-prep branch from 1f17b84 to 70275a3 Compare May 9, 2023 01:15

arnocandel added 2 commits May 8, 2023 18:21

Add row_id for oig_oasst1 and oasst1 and fortune2000 data.

f4e037d

Fix get_sentences.

d1f8ce9

arnocandel marked this pull request as ready for review May 9, 2023 03:09

arnocandel merged commit 332e272 into main May 9, 2023

This pull request was closed.

Provide feedback