Summary of the most important changes:
- Slowly rotating pool of cached samples.
- Up to 20k samples and losses are cached for all available models
- Much faster step times: models in the pools typically don't have to be evaluated over and over on new samples.
- Refresh rate: 500 samples per 4 hours.
- Increase samples/step to 1500 (was 800) for more fine-grained model comparison
- Remove logging of selected sample pages, because the sample pool should remain private to the validator.
- Adjusted model re-evaluation criteria:
- All models: once every 48 hours (if not marked as non-competitive when wins and abs wins <2%)|
- Top models: once every 8 hours
- Top model criterion: one of top 10 validators >=0.05 weight (was: 0.1)
- Duplicate commitments of the same repo are filtered out
- Model content duplications are filtered out
- Sync/download thread obeys a TTL so that top and new models have priority over re-evaluation of other models
- Improved logging about which uids/models are visited