initial data trends and analysis

[pszemraj]

initial data trends and analysis

this is a discussion to post/discuss etc about trends/analysis for the initial version (release v0.1.1):

example topics

Some initial ideas of things to explore, discuss, and investigate:

• Are any of the "automatic" writing quality metrics useful? Are any not?
  • Do you notice any bias or confounding variables? For example, models that produce technical/formal sounding summaries seem to typically be rated as "better written" when this may not be the case.
• Is there any point in using/calculating the CoLA and gibberish predictions as a means of filtering out/excluding models?
  • Are summaries predicted to have poor CoLA/gibberish scores actually bad?
• Data cleaning:** - what else needs to be done? which of the columns can/should be dropped (there are probably many)? which should be mapped/consolidated?

Writing Quality - some details

Some additional details/exports:

• this is the output of microsoft/deberta-v2-xlarge-mnli running zero-shot writing quality inference
• the plot just shows the median score for all parameters/runs by model
• interactive plot link

[pszemraj]

WQ data for deberta-v2-xlarge-mnli

this data is already under the releases for tag v0.1.1 but putting it here too:

summary_gauntlet_dataset_mapped_src_docs_WQ_predictionsdeberta-v2-xlarge-mnli.csv

summary_gauntlet_dataset_mapped_src_docs_WQ_predictionsdeberta-v2-xlarge-mnli.zip

[pszemraj]

token lengths

these are for the summary column

Name: summary_tokens_long-t5-tglobal-base, dtype: float64

count    2043.000000
mean      524.828683
std       773.441398
min         9.000000
25%       115.000000
50%       258.000000
75%       614.500000
max      7736.000000

Name: summary_tokens_pythia-2.8b, dtype: float64

count    2043.000000
mean      478.701419
std       698.122262
min         7.000000
25%       105.000000
50%       237.000000
75%       563.500000
max      6948.000000

Name: summary_tokens_deberta-large-mnli, dtype: float64

count    2043.000000
mean      479.813020
std       692.595392
min         7.000000
25%       107.000000
50%       238.000000
75%       565.000000
max      6888.000000