some tests

[yiftachashkenazi]

Section-by-Section Word Count Analysis

Section	Avg Words	Target Range	In Range %	Too Short %	Too Long %	Avg Utilization %
Research Strategy	1099.7	1444-1806	0.0%	100.0%	0.0%	60.9%
Preliminary Results	332.3	288-361	100.0%	0.0%	0.0%	92.1%
Abstract	229.0	228-285	66.7%	33.3%	0.0%	80.4%

Word Count Distribution Summary

• Overall In Range: 5/9 (55.6%)
• Overall Too Short: 4/9 (44.4%)
• Overall Too Long: 0/9 (0.0%)

ROUGE Metrics Analysis

Overall ROUGE Statistics

Metric	Average	Min	Max	Std Dev
ROUGE-1	0.2786 (27.9%)	0.2668	0.2882	0.0109
ROUGE-2	0.0886 (8.9%)	0.0830	0.0995	0.0094
ROUGE-L	0.1463 (14.6%)	0.1413	0.1522	0.0055
ROUGE-SU	0.2216 (22.2%)	0.2117	0.2266	0.0085

ROUGE Interpretation

• ROUGE-1: 27.9% word overlap with RAG sources
• ROUGE-2: 8.9% bigram overlap (phrase similarity)
• ROUGE-L: 14.6% longest sequence preservation
• ROUGE-SU: 22.2% skip-bigram + unigram combination

Generated Words for ROUGE Tests

• Average: 223.3 words
• Range: 207 - 243 words
• Standard Deviation: 18.2

Key Findings

Word Count Performance

• Overall Performance: (55.6% sections in target range)

ROUGE Performance

• Overall Performance: 🟡 Good RAG utilization (0.1838 average score)
• Content Utilization: 27.9% of RAG content utilized in generation
• Originality: 91.1% original content (low bigram overlap indicates good originality)

[yiftachashkenazi]

@Goldziher fyi, results of two test I added for better control of section writing part