Skip to content

Unit tests with network calls cause flaky failures (504 timeouts) #253

@timenick

Description

@timenick

Problem

Several test files under tests/unit/ make real network calls to HuggingFace Hub, causing flaky CI failures when HF is slow or unavailable (e.g., 504 Gateway Timeout).

Affected files

  1. tests/unit/eval/test_text_classification.py — Every test instantiates TextDataset which calls datasets.load_dataset() to download GLUE/MRPC from HuggingFace Hub.

  2. tests/unit/loader/test_hf_model_class_mapping.pyTestResolveTaskAndModelClass (9 tests) and TestConflictScenarios (5 tests) call AutoConfig.from_pretrained() which downloads model configs from HuggingFace Hub. These were missed in Move misplaced integration/e2e tests out of tests/unit/ #95 which moved the TestHFModelClassMappingE2E class but left the AutoConfig.from_pretrained() callers behind.

Impact

  • Unit tests should be fast, deterministic, and offline
  • Network failures cause spurious test failures (observed: HfHubHTTPError: 504 Server Error)
  • Slows down the unit test suite unnecessarily

Fix

Move these tests to tests/integration/ where network-dependent tests belong.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions