Data synthesis

[mmirman]

We should use the larger LLM to synthesize data for training the small LLM with in the optimizing api

[abhigya-sodani]

#10

[cartazio]

#12 mentions some features this should have

1. duplicate/ too similar a response detection

duplicates can be handled with a hashing based lookup, or even maybe something like a bloom filter if we didn't want much. any strong hash function would work for duplicate detection and then its just a question of how much space overhead we're ok with.

@abhigya-sodani proposed a nice way to approach "too similar responses". namely the Euclidean aka L2 aka innerproduct aka cosine similarity distance of pairs of responses. This does run into the issue of having a quadratic blow up in operations as a function of the number of examples. One approach to resolve that would be to maintain some approximate nearest neighbors data structure and only compare against the closest few. I'm very unfamiliar with the current state of the art for ANN tools, let alone which ones have a quality python interface, @mmirman any strong opinions there?

[mmirman]

This is already an approximate measure of dataset diversity so why not just roll with that and try to make it a probabilistic approximate measure of dataset diversity? Like take a subsample of random pairs and compute diversity of that set.

[mmirman]

Or a few sub-cliques. I'm sure there's some statistically optimal way of doing this that doesn't matter since we aren't publishing

[mmirman]

bloom filter is overkill until we have users complaining

[cartazio]

This is already an approximate measure of dataset diversity so why not just roll with that and try to make it a probabilistic approximate measure of dataset diversity? Like take a subsample of random pairs and compute diversity of that set.

you mean that average L2 norm of the pairs of distances? eg average_i=0,n, j>i of ( dist(i,j) )

ok cool, i'm comfortable with the subsampling then doing the all pairs average. (not where I was gonna go, but a good place none the less).

[cartazio]

wrt embedding vector computation text-embedding-ada-002 would be the model invoked in the preliminary version I suppose, at least for the openai flavored approach, right?

[cartazio]

though thats not an ideal end state

[mmirman]

average is probably the wrong statistic but something like itBest,Matt==================Dr. Matthew Mirmanhttps://www.mirman.comOn 27 Jun 2023, at 8:20 PM, Carter Tazio Schonwald ***@***.***> wrote: This is already an approximate measure of dataset diversity so why not just roll with that and try to make it a probabilistic approximate measure of dataset diversity? Like take a subsample of random pairs and compute diversity of that set. you mean that average L2 norm of the pairs of distances? eg average_i=0,n, j>i of ( dist(i,j) ) ok cool, i'm comfortable with the subsampling then doing the all pairs average. (not where I was gonna go, but a good place none the less). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[mmirman]

like, avg # of pairs sampled before a pair of distance below X?Best,Matt==================Dr. Matthew Mirmanhttps://www.mirman.comOn 27 Jun 2023, at 8:24 PM, Carter Tazio Schonwald ***@***.***> wrote: though thats not an ideal end state —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[cartazio]

oh, like "min and 10th percentile distance" among subsampled pairs?

[cartazio]

recap: exact duplicate detection, then feature vector comparisons

[cartazio]

items from #24 are

• Add all the parameters
• Prompt variation aka support alternative prompts, possibly with different defaults for each supported model
• Track number of actual responses vs requested and other possible statistics on quality
• Dedup handling
  -[ ]First exact repeat duplicate detection
  -[ ] then semantic vector comparisons for duplicate detection (this is going to possibly be a bit more fiddly)
  -[ ] consider adding Parameters for the diverse ways you might want to define exact matches or vector comparisons

[cartazio]

#29 is also important

[cartazio]

632538f added basic duplicate detection, but theres a LOT more work to be done there, with the next future step being evaluating Spacy and OpenAI/LLM semantic vector embeddings and parameter tuning around that

[cartazio]

still needs K-shot example support