main points 15/03 meeting #51
rbroc
started this conversation in
Meeting Notes
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Mina has done some really nice work to finalize data generation! Here is a record of the main points discussed during our 15/03 meeting and next steps
Prompts finalized
Prompts kept simple and general. The final list is here: https://github.com/rbroc/echo/blob/main/src/generate/prompts.py#L52-56
As mentioned in #47, we are running with no system prompt for LLama. For stories, one of the two prompts will be selected (as far as I remember, the first of the two -- but @MinaAlmasi correct me if I am wrong).
Decoding parameters finalized
Mina has generated data with temperature = 1.0, 1.5 and 2.0. A lot of the 2.0 data makes no sense -- hence not planning on using this data for the rest of the project, really. We will work with 1.0 and 1.5, and later decide which dataset we sample from for the human experiment.
Data cleaning
There are a few cases where models don't follow instructions, or start generating nonsensical stuff. They are probably not many, and they occur more with some models (e.g., LLaMa) and some tasks (stories) than others. Filtering this data in a rule-based way seems pretty much impossible. A possible approach to removing this kind of data would be few-shot learning using
SetFit
on some manual "quality" annotations. On the other hand, these are not that many, and we can also consider these failure scenarios as "signals" that classifiers and humans can use to detect whether text is AI or human generated. Therefore, we lean towards not excluding any data ATM, and possibly reconsidering our decision later on. But maybe we can exclude clear failure cases from the human experiment though.Note, however, that at some point it could be nice to annotate the data for whether they are "good" completions (grammatical & following instructions) or not anyway. These annotations could actually be relevant for:
Next steps
Classify!
Mina will rerun generation for one of the datasets where the max tokens was specified incorrectly, and then we're ready to train a classifier. :)
Raw features vs PCA
For now, we will work with raw TextDescriptives features and probably with XGBoost/RandomForest. However, features are likely to be highly correlated and we may therefore consider PCA-ing our way out of multicollinearity. Multicollinearity won't be a problem for predictive performance, but it might for the interpretation of feature importances. Though perhaps less so if we are conservative in terms of tree complexity (low feature bagging params + low tree depth).
How many models?
I'd probably lean towards training separate classifiers for each task (and temperature), but the same classifier for multiple LLMs.
Open questions
@rdkm89 and @MinaAlmasi, feel free to add / edit if needed :)
since we're starting to dive into the really interesting stuff, let's try to have a meeting in the coming weeks with the whole group.
Beta Was this translation helpful? Give feedback.
All reactions