Analysis date: 2026-05-19
Total records: 32,759,915
Dataset: Code.org Computer Science Discoveries Unit 3 (CSD3)
- Methodology
- Temporal Structure of the Dataset
- RQ1 — Time-of-Day Context: Classroom vs. Homework
- RQ2 — Seasonality: School Year vs. Summer
- Statistical Test Summary
- Discussion & Limitations
| Question | |
|---|---|
| RQ1 | Do students engaging with CSD3 during school hours (classroom context) perform differently from students working after school (homework context)? |
| RQ2 | Does student performance and engagement differ between the school year (Sep–May) and summer break (Jun–Aug)? |
Time-of-Day (RQ1) — based on UTC timestamps:
| Context | Definition | Rationale |
|---|---|---|
| Classroom | Weekday (Mon–Fri), hours 12–20 UTC | ≈ 7 am–3 pm EST / 4 am–noon PST — US school day |
| Homework | All other times | Weekday evenings, early mornings, all weekend |
Seasonality (RQ2):
| Season | Months | Definition |
|---|---|---|
| School Year | Sep–Dec, Jan–May (months 1–5, 9–12) | US academic calendar |
| Summer | Jun–Aug (months 6–8) | US summer break |
| Group | Records | MC Records | Unique Students |
|---|---|---|---|
| Classroom | 28,522,305 (87.1%) | 1,397,633 | 268,237 |
| Homework | 4,237,610 (12.9%) | 226,097 | 118,579 |
| School Year | 31,923,736 (97.4%) | 1,568,854 | 277,331 |
| Summer | 836,179 (2.6%) | 54,876 | 31,943 |
A single student may appear in multiple groups (e.g., classroom on Tuesday and homework on Sunday).
A level interaction is successful if best_result ∈ {100, 101} (codes for all_pass and contained_level_result).
Six hypotheses were tested (3 for RQ1, 3 for RQ2). A Bonferroni correction gives α* = 0.05 / 6 ≈ 0.0083.
| Test type | Used for | Effect size metric |
|---|---|---|
| Two-proportion z-test | Success rate differences | Cohen's h |
| Mann–Whitney U (50k samples) | Attempts & time distributions | Rank-biserial r |
Key observations:
- 87.1% of all records fall in the classroom window (weekdays 12–20 UTC). The data is overwhelmingly school-day driven.
- Weekend activity is minimal: Saturday (220,877) + Sunday (219,210) = just 1.3% of records.
- The UTC peak at 13–18h on weekdays matches the Eastern Time school day (8 am–1 pm), confirming the classroom signal.
- Summer contributes only 2.6% of total records, concentrated in August (likely pre-course preparation) and June.
Classroom students answered MC questions correctly on their first try 38.9% of the time, vs. 32.0% for homework students — a 6.9 percentage point advantage.
| Metric | Classroom | Homework | Δ |
|---|---|---|---|
| First-attempt success (MC) | 38.9% | 32.0% | +6.9 pp |
| Overall level success (all types) | 10.7% | 9.0% | +1.7 pp |
| Median time per level | 104 s | 106 s | −2 s |
| Median attempts per level | 3 | 3 | 0 |
| Mean attempts | 7.67 | 8.36 | −0.69 |
Interpretation: Classroom students come better prepared — they succeed more on first try — but do not work harder or longer afterward. The reattempt behavior is identical.
The 2-second difference in median time (104s vs 106s) is statistically significant (p = 1.3×10⁻⁶) but the rank-biserial r = 0.018 is negligible. Classroom students are marginally faster, possibly due to teacher guidance reducing exploration time.
School year students succeed on MC first attempts 38.1% of the time vs. 33.7% in summer — a 4.4 pp gap (z = 42.5, p < 10⁻³⁰⁰, Cohen's h = 0.091).
| Metric | School Year | Summer | Δ |
|---|---|---|---|
| First-attempt success (MC) | 38.1% | 33.7% | +4.4 pp |
| Overall level success | 10.5% | 9.5% | +1.0 pp |
| Median time per level | 105 s | 90 s | +15 s |
| Mean time per level | 231.8 s | 208.7 s | +23.1 s |
| MC records on first attempt | ~83% | ~45% | — |
Summer students retry MC questions far more often — only ~45% of summer MC records are first attempts vs ~83% during the school year. Summer users behave more exploratorily: they try, fail, and try again, while school-year students mostly attempt a question once and move on.
Summer students spend significantly less time per level (median 90s vs 105s, p = 1.6×10⁻³⁷, r = −0.047). Combined with the more multi-attempt behavior, this suggests summer learners are engaging in shorter, more rapid-fire sessions rather than sustained focused work.
Success rates are remarkably stable across the school year (~10–11%), with a modest dip in summer. August shows the lowest summer success rate (9.2%), consistent with self-directed pre-course learners encountering material cold. July shows an anomalous 11.8% success rate — likely a small, self-selected population of motivated summer learners.
| Hypothesis | Question | Test | Statistic | p-value | Effect Size | Significant? |
|---|---|---|---|---|---|---|
| H1a | First-attempt success: classroom vs. homework | Two-proportion z-test | z = 140.97 | < 10⁻³⁰⁰ | Cohen's h = 0.143 (small) | ★ Yes |
| H1b | Attempts distribution: classroom vs. homework | Mann–Whitney U | U = 1.25×10⁹ | p = 0.3899 | r = 0.003 (negligible) | No |
| H1c | Time per level: classroom vs. homework | Mann–Whitney U | U = 1.23×10⁹ | p = 1.3×10⁻⁶ | r = 0.018 (negligible) | ★ Yes |
| H2a | First-attempt success: school year vs. summer | Two-proportion z-test | z = 42.55 | < 10⁻³⁰⁰ | Cohen's h = 0.091 (small) | ★ Yes |
| H2b | Attempts: school year vs. summer | Mann–Whitney U | U = 1.28×10⁹ | p = 8.8×10⁻¹¹ | r = −0.023 (small) | ★ Yes |
| H2c | Time per level: school year vs. summer | Mann–Whitney U | U = 1.31×10⁹ | p = 1.6×10⁻³⁷ | r = −0.047 (small) | ★ Yes |
Bonferroni-corrected threshold: α* = 0.05 / 6 = 0.0083
4 of 6 hypotheses are significant. The non-significant result (H1b) is notable: classroom and homework students do not differ in how hard they try after a first failure.
Classroom students succeed on MC first attempts 6.9 pp more often — a small but consistent effect at this scale. Yet the number of attempts is statistically indistinguishable. This pattern is consistent with teacher-mediated instruction providing better pre-attempt scaffolding: students in class come more prepared, not more persistent.
The time difference (2 s median) is significant only because of the massive sample size. It is practically meaningless.
Summer represents a qualitatively different engagement mode:
- Less time per session (90s vs 105s median) — shorter, more rapid interactions
- More reattempts on MC questions (~45% first-attempt records vs ~83%) — more exploratory
- Lower first-attempt success (33.7% vs 38.1%) — possibly reflecting self-directed learners without instructional support
The summer users are likely a self-selected, more motivated population (voluntarily engaging without class requirements), which may suppress how large the performance gap would otherwise be.
- Timezone confound: All timestamps are UTC. West Coast (PST) students in the 12–15 UTC window are counted as "classroom" but may be working at 4–7 am. The classroom window is calibrated for Eastern Time.
- Ecological design: The classroom/homework split is inferred from timestamps, not experimental. Unmeasured confounders (teacher quality, prior knowledge, device access) likely explain part of the effect.
- Non-independence: A single student can appear in multiple groups on different days.
- Summer selection bias: Summer users may be disproportionately motivated self-learners. The lower success rate may understate the true summer deficit for a representative population.
- Level-type composition: Different level types (Applab, Multi, etc.) may be emphasized at different times of year, confounding seasonal comparisons.
Analysis: Code.org CSD3 2023–2025 · 32,759,915 records · Generated 2026-05-19