Skip to content

Commit

Permalink
pull assemblies into benchmarks; add versioning; add public benchmarks (
Browse files Browse the repository at this point in the history
#175)

* separate benchmarks by their respective assemblies

* move assembly loading into respective benchmark files

* add versions

* separate unit tests according to new benchmark/assembly structure

* separate unit tests according to new benchmark/assembly structure

* rename behavior -> image_level_behavior (contains I2n)

* work around missing neuroid dimension to to xarray collapsing multiindex

* add method to list assemblies that user has access to

* get rid of overly verbose logging; compare sets of assemblies

* add public benchmarks for V1, V2, V4, IT, match-to-sample

* give more leeway for Cadena benchmark

rationale: we don't use this benchmark much anyway and I'm not sure when we have last tested this score

* get rid of temporal predictivity benchmarks

rationale: these benchmarks are WIP and until we agree on the right metric and ceiling, let's not put them in the Brain-Score standards

* rename test_behavioral -> test_rajalingham2018

* defer __len__ and __class__ to LazyLoad's content

* fix README header

* add unit tests with precomputed model features

* add version property to Benchmark interface

* use public assemblies and benchmarks in examples

* fix separate stratification_coord; add unittests for public benchmarks

* remove private_access mark since examples now only use public assemblies

* fix V4 precomputed score; fix ceiling averaging

* add imagenet to public benchmarks; remove specific movshon benchmark in list
  • Loading branch information
mschrimpf committed Dec 9, 2019
1 parent 8302002 commit d03e50f
Show file tree
Hide file tree
Showing 42 changed files with 847 additions and 1,209 deletions.
96 changes: 0 additions & 96 deletions brainscore/assemblies/__init__.py

This file was deleted.

254 changes: 0 additions & 254 deletions brainscore/assemblies/private.py

This file was deleted.

0 comments on commit d03e50f

Please sign in to comment.