Skip to content

v1.0.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 03 May 12:47
· 1 commit to main since this release
v1.0.0
2da57a1

v1.0.0 (2024-05-03)

Breaking

  • chore: empty commit to bump release

BREAKING CHANGE: v1 release (2615a3e)

Chore

  • chore: fix outdated lr_scheduler_name in docs (decoderesearch#109)

  • chore: fix outdated lr_scheduler_name in docs

  • add tutorial hparams (7cba332)

  • chore: enabling pythong 3.12 checks for CI (25526ea)

  • chore: setting up precommit to be consistent with CI (18e706d)

  • chore: re-enabling isort in CI (decoderesearch#86) (9c44731)

  • chore: improving CI speed (9e3863c)

  • chore: updating README.md with pip install instructions and PyPI badge (682db80)

  • chore: adding more tests to ActivationsStore + light refactoring (cc9899c)

  • chore: running isort to fix imports (53853b9)

  • chore: setting up pyright type checking and fixing typing errors (351995c)

  • chore: enable full flake8 default rules list (19886e2)

  • chore: using poetry for dependency management (465e003)

  • chore: removing .DS_Store files (32f09b6)

Feature

  • feat: add activation bins for neuronpedia outputs, and allow customizing quantiles (decoderesearch#113) (05d650d)

  • feat: Update for Neuropedia auto-interp (decoderesearch#112)

  • cleanup Neuronpedia autointerp code

  • Fix logic bug with OpenAI key

Co-authored-by: Joseph Bloom <69127271+jbloomAus@users.noreply.github.com> (033283d)

  • feat: SparseAutoencoder.from_pretrained() similar to transformer lens (decoderesearch#111)

  • add partial work so David can continue

  • feat: adding a SparseAutoencoder.from_pretrained() function

Co-authored-by: jbloomaus <jbloomaus@gmail.com> (617d416)

  • feat: breaks up SAE.forward() into encode() and decode() (decoderesearch#107)

  • breaks up SAE.forward() into encode() and decode()

  • cleans up return typing of encode by splitting into a hidden and public function (7b4311b)

  • feat: API for generating autointerp + scoring for neuronpedia (decoderesearch#108)

  • API for generating autointerp for neuronpedia

  • Undo pytest vscode setting change

  • Fix autointerp import

  • Use pypi import for automated-interpretability (7c43c4c)

  • feat: make a neuronpedia list with features via api call (decoderesearch#101) (23e680d)

  • feat: Added tanh-relu activation fn and input noise options (decoderesearch#77)

  • Still need to pip-install from GitHub hufy implementation.

  • Added support for tanh_sae.

  • Added notebook for loading the tanh_sae

  • tweaking config options to be more declarating / composable

  • testing adding noise to SAE forward pass

  • updating notebook

Co-authored-by: David Chanin <chanindav@gmail.com> (551e94d)

  • feat: Mamba support vs mamba-lens (decoderesearch#79)

  • mamba support

  • added init

  • added optional model kwargs

  • Support transformers and mamba

  • forgot one model kwargs

  • failed opts

  • tokens input

  • hack to fix tokens, will look into fixing mambalens

  • fixed checkpoint

  • added sae group

  • removed some comments and fixed merge error

  • removed unneeded params since that issue is fixed in mambalens now

  • Unneded input param

  • removed debug checkpoing and eval

  • added refs to hookedrootmodule

  • feed linter

  • added example and fixed loading

  • made layer for eval change

  • fix linter issues

  • adding mamba-lens as optional dep, and fixing typing/linting

  • adding a test for loading mamba model

  • adding mamba-lens to dev for CI

  • updating min mamba-lens version

  • updating mamba-lens version

Co-authored-by: David Chanin <chanindav@gmail.com> (eea7db4)

  • feat: support orthogonal decoder init and no pre-decoder bias (ac606a3)

  • feat: add basic tutorial for training saes (1847280)

  • feat: overhaul saving and loading (004e8f6)

  • feat: release (c70b148)

Fix

  • fix: replace list_files_info with list_repo_tree (decoderesearch#117) (676062c)

  • fix: Improved activation initialization, fix using argument to pass in API key (decoderesearch#116) (7047bcc)

  • fix: pin pyzmq==26.0.1 temporarily (0094021)

  • fix: typing issue, temporary (25cebf1)

  • fix: sae dict bug (484163e)

  • fix: session loader wasn't working (a928d7e)

  • fix: dense batch dim mse norm optional (8018bc9)

  • fix: neuronpedia quicklist (6769466)

  • fix: removing paths-ignore from action to avoid blocking releases (28ff797)

  • fix: updating saevis version to use pypi (dbd96a2)

Unknown

  • BREAKING CHANGE: 1.0.0 release

BREAKING CHANGE: 1.0.0 release (c23098f)

Improvements to Neuronpedia Runner (5118f7f)

  • neuronpedia: save run settings to json file to avoid errors when resuming later. automatically skip batch files that already exist (4b5412b)

  • skip batch file if it already exists (7d0e396)

  • neuronpedia: include log sparsity threshold in skipped_indexes.json (5c967e7)

  • Update proposal.md (6d45b33)

  • Merge pull request decoderesearch#96 from jbloomAus/github-templates

add templates for PR's / issues (241a201)

  • add templates for PR's / issues (74ff597)

  • Merge pull request decoderesearch#95 from jbloomAus/load-state-dict-not-strict

Make load_state_dict use strict=False (4a9e274)

  • fix accidental bug (c22fbbd)

  • fix load pretrained legacy with state dict change (b5e97f8)

  • Make load_state_dict use strict=False (fdf7fe9)

  • Merge pull request decoderesearch#94 from jbloomAus/update-pre-commit

chore: setting up precommit to be consistent with CI (6a056b7)

Adds function that converts old .pt pretrained SAEs to new folder format (1cb1725)

chore: enabling python 3.12 checks for CI (87be422)

  • v0.5.1 (0ac218b)

  • fixes string vs path typing errors (94f1fc1)

  • removes unused import (06406b0)

  • updates formatting for alignment with repo standards. (5e1f342)

  • consolidates with SAE class load_legacy function & adds test (0f85ded)

  • adds old->new file conversion function (fda2b57)

  • Merge pull request decoderesearch#91 from jbloomAus/decoder-fine-tuning

Decoder fine tuning (1fc652c)

Enhance + Fix Neuronpedia generation / upload (38d507c)

FEAT: Add API for getting Neuronpedia feature (1666a68)

  • Fix resuming from batch (145a407)

  • Use original repo for sae_vis (1a7d636)

  • Use correct model name for np runner (138d5d4)

  • Merge main, remove eindex (6578436)

  • Add API for getting Neuronpedia feature (e78207d)

  • update readme (440df7b)

  • update readme (3694fd2)

  • Fix upload skipped/dead features (932f380)

  • Use python typer instead of shell script for neuronpedia jobs (b611e72)

  • Merge branch 'main' into fix_np (cc6cb6a)

  • convert sparsity to log sparsity if needed (8d7d404)

  • enable setting adam pars in config (1e53ede)

  • fix sae dict loader and format (c558849)

  • default orthogonal init false (a8b0113)

  • Formatting (1e3d53e)

  • Eindex required by sae_vis (f769e7a)

  • Upload dead feature stubs (9067380)

  • Make feature sparsity an argument (8230570)

  • Fix buffer" (dde2481)

  • Merge branch 'main' into fix_np (6658392)

  • notebook update (feca408)

  • Merge branch 'main' into fix_np (f8fb3ef)

  • Final fixes (e87788d)

  • Don't use buffer, fix anomalies (2c9ca64)

  • format (c359c27)

  • make dense_batch_mse_normalization optional (c41774e)

  • Runner is fixed, faster, cleaned up, and now gives whole sequences instead of buffer. (3837884)

  • Merge branch 'main' into fix_np (3ed30cf)

  • add warning in run script (9a772ca)

  • update sae loading code (356a8ef)

  • add device override to session loader (96b1e12)

  • update readme (5cd5652)

  • Use legacy loader, add back histograms, logits. Fix anomaly characters. (ebbb622)

  • Merge branch 'main' into fix_np (586e088)

  • Merge pull request decoderesearch#80 from wllgrnt/will-update-tutorial

bugfix - minimum viable updates to tutorial notebook (e51016b)

perf: improving CI speed (8b00000)

  • try partial cache restore (392f982)

  • Merge branch 'main' into faster-ci (89e1568)

  • Merge pull request decoderesearch#78 from jbloomAus/fix-artifact-saving-loading

Fix artifact saving loading (8784c74)

  • remove duplicate code (6ed6af5)

  • set device in load from pretrained (b4e12cd)

  • fix typing issue which required ignore (a5df8b0)

  • remove print statement (295e0e4)

  • remove load with session option (74926e1)

  • fix broken test (16935ef)

  • avoid tqdm repeating during training (1d70af8)

  • avoid division by 0 (2c7c6d8)

  • remove old notebook (e1ad1aa)

  • use-sae-dict-not-group (27f8003)

  • formatting (827abd0)

  • improve artifact loading storage, tutorial forthcoming (604f102)

  • add safetensors to project (0da48b0)

  • Don't precompute background colors and tick values (271dbf0)

  • Merge pull request decoderesearch#71 from weissercn/main

Addressing notebook issues (8417505)

chore: updating README.md with pip install instructions and PyPI badge (4d7d1e7)

  • FIX: Add back correlated neurons, frac_nonzero (d532b82)

  • linting (1db0b5a)

  • fixed graph name (ace4813)

  • changed key for df_enrichment_scores, so it can be run (f0a9d0b)

  • fixed space in notebook 2 (2278419)

  • fixed space in notebook 2 (24a6696)

  • fixed space in notebook (d2f8c8e)

  • fixed pickle backwards compatibility in tutorial (3a97a04)

  • Merge pull request decoderesearch#69 from chanind/remove-ci-ignore

fix: removing paths-ignore from action to avoid blocking releases (179cea1)

fix: hotfix updating saevis version to use pypi (a13cee3)

feat: setting up sae_lens package and auto-deploy with semantic-release (34633e8)

fixing config typing (b8fba4f)

  • setting up sae_lens package and auto-deploy with semantic-release (ba41f32)

  • fixing config typing

switch to using explicit params for ActivationsStore config instead of RunnerConfig base class (9be3445)

passing accidentally overlooked scheduler opts (773bc02)

adding lr_decay_steps and refactoring get_scheduler (c960d99)

  • adding lr_decay_steps and refactoring get_scheduler (fd5448c)

  • Merge pull request decoderesearch#53 from hijohnnylin/neuronpedia_runner

Generate and upload Neuronpedia artifacts (0b94f84)

removing unused eindex depencency (1ce44d7)

  • removing unused eindex depencency (7cf991b)

  • Safe to_str_tokens, fix memory issues (901b888)

  • Allow starting neuronpedia generation at a specific batch numbe (85d8f57)

  • FIX: Linting 'do not use except' (ce3d40c)

  • Fix vocab: Ċ should be line break. Also set left and right buffers (205b1c1)

  • Merge (b159010)

  • Update Neuronpedia Runner (885de27)

  • Merge pull request decoderesearch#58 from canrager/main

Make prepend BOS optional: Default True (48a07f9)

attempting to fix docs deploy (cfafbe7)

Adding tests to get_scheduler (13c8085)

minor refactoring to SAE and adding tests (2c425ca)

  • minor refactoring to SAE and adding tests (92a98dd)

  • adding tests to get_scheduler (3b7e173)

  • Generate and upload Neuronpedia artifacts (b52e0e2)

  • Merge pull request decoderesearch#54 from jbloomAus/hook_z_suppourt

notional support, needs more thorough testing (277f35b)

adding a contribution guide to docs (8ac8f05)

  • adding a contribution guide to docs (693c5b3)

  • notional support, needs more thorough testing (9585022)

  • Generate and upload Neuronpedia artifacts (4540268)

  • Merge pull request decoderesearch#52 from hijohnnylin/fix_db_runner_assert

FIX: Don't check wandb assert if not using wandb (5c48811)

Add Docs to the project. (d2ebbd7)

Pin sae_vis to previous working version (3f8a30b)

Refactor train SAE and adding unit tests (5aa0b11)

  • Merge branch 'main' into refactor-train-sae (0acdcb3)

  • Merge pull request decoderesearch#41 from jbloomAus/move_to_sae_vis

Move to sae vis (bcb9a52)

chore: adding more tests to ActivationsStore + light refactoring (69dcf8e)

  • Merge branch 'main' into activations_store_tests (4896d0a)

  • refactoring train_sae_on_language_model.py into smaller functions (e75a15d)

  • suppourt apollo pretokenized datasets (e814054)

  • handle saes saved before groups (5acd89b)

  • typechecker (fa6cc49)

  • fix geom median bug (8d4a080)

  • remove references to old code (861151f)

  • remove old geom median code (05e0aca)

  • Merge pull request decoderesearch#22 from themachinefan/faster_geometric_median

Faster geometric median. (341c49a)

  • makefile check type and types of geometric media (736bf83)

  • Merge pull request decoderesearch#21 from schmatz/fix-dashboard-image

Fix broken dashboard image on README (eb90cc9)

Added link to AF post (39f8d3d)

chore: setting up pyright type checking and fixing typing errors (bd5fc43)

SAE Group for sweeps PR (3e78bce)

  • Merge pull request #1 from chanind/sae_group_pr_isort_fix

chore: running isort to fix imports (dd24413)

chore: enable full flake8 default rules list (ad84706)

chore: using poetry for dependency management (496f7b4)

  • progress on implementing multi-sae support (2ba2131)

  • Merge pull request decoderesearch#11 from lucyfarnik/fix-caching-shuffle-edge-case

Fixed edge case in activation cache shuffling (3727b5d)

Added run name to config (c2e05c4)

  • Added run name to config (ab2aabd)

  • Fixed edge case in activation cache shuffling (18fd4a1)

  • Merge pull request decoderesearch#9 from chanind/rm-ds-store

chore: removing .DS_Store files (37771ce)

  • improve readmen (f3fe937)

  • fix_evals_bad_rebase (22e415d)

  • evals changes, incomplete (736c40e)

  • make tutorial independent of artefact and delete old artefact (6754e65)

  • fix MSE in ghost grad (44f7988)

  • Merge pull request decoderesearch#5 from jbloomAus/clean_up_repo

Add CI/CD, black formatting, pre-commit with flake8 linting. Fix some bugs. (01ccb92)

  • clean up run examples (9d46bdd)

  • move where we save the final artifact (f445fac)

  • fix activations store innefficiency (07d38a0)

  • black format and linting (479765b)

  • dummy file change (912a748)

  • try adding this branch listed specifically (7fd0e0c)

  • yml not yaml (9f3f1c8)

  • add ci (91aca91)

  • get units tests working (ade2976)

  • make unit tests pass, add make file (08b2c92)

  • add pytest-cov to requirements.txt (ce526df)

  • seperate research from main repo (32b668c)

  • remove comma and set default store batch size lower (9761b9a)

  • notebook for Johny (39a18f2)

  • best practices ghost grads fix (f554b16)

  • Update README.md

improved the hyperpars (2d4caf6)

Ghost grads dev (4d150c2)

  • save final log sparsity (98e4f1b)

  • start saving log sparsity (4d6df6f)

  • get ghost grads working (e863ed7)

  • add notebook/changes for ghost-grad (not working yet) (73053c1)

  • idk, probs good (0407ad9)

  • bunch of shit (1ec8f97)

  • Merge branch 'main' of github.com:jbloomAus/mats_sae_training (a22d856)

  • Reverse engineering the "not only... but" feature (74d4fb8)

  • Merge pull request #2 from slavachalnev/no_reinit

Allow sampling method to be None (4c5fed8)

  • Allow sampling method to be None (166799d)

  • research/week_15th_jan/gpt2_small_resid_pre_3.ipynb (52a1da7)

  • add arg for dead neuron calc (ffb75fb)

  • notebooks for lucy (0319d89)

  • add args for b_dec_init (82da877)

  • add geom median as submodule instead (4c0d001)

  • add geom median to req (4c8ac9d)

  • add-geometric-mean-b_dec-init (d5853f8)

  • reset feature sparsity calculation (4c7f6f2)

  • anthropic sampling (048d267)

  • get anthropic resampling working (ca74543)

  • add ability to finetune existing autoencoder (c1208eb)

  • run notebook (879ad27)

  • switch to batch size independent loss metrics (0623d39)

  • track mean sparsity (75f1547)

  • don't stop early (44078a6)

  • name runs better (5041748)

  • improve-eval-metrics-for-attn (00d9b65)

  • add hook q (b061ee3)

  • add copy suppression notebook (1dc893a)

  • fix check in neuron sampling (809becd)

  • Merge pull request #1 from jbloomAus/activations_on_disk

Activations on disk (e5f198e)

  • merge into main (94ed3e6)

  • notebook (b5344a3)

  • various research notebooks (be63fce)

  • Added activations caching to run.ipynb (054cf6d)

  • Added activations dir to gitignore (c4a31ae)

  • Saving and loading activations from disk (309e2de)

  • Fixed typo that threw out half of activations (5f73918)

  • minor speed improvement (f7ea316)

  • add notebook with different example runs (c0eac0a)

  • add ability to train on attn heads (18cfaad)

  • add gzip for pt artefacts (9614a23)

  • add_example_feature_dashboard (e90e54d)

  • get_shit_done (ce73042)

  • commit_various_things_in_progress (3843c39)

  • add sae visualizer and tutorial (6f4030c)

  • make it possible to load sae trained on cuda onto mps (3298b75)

  • reduce hist freq, don't cap re-init (debcf0f)

  • add loader import to readme (b63f14e)

  • Update README.md (88f086b)

  • improve-resampling (a3072c2)

  • add readme (e9b8e56)

  • fixl0_plus_other_stuff (2f162f0)

  • add checkpoints (4cacbfc)

  • improve_model_saving_loading (f6697c6)

  • stuff (19d278a)

  • Added support for non-tokenized datasets (afcc239)

  • notebook_for_keith (d06e09b)

  • fix resampling bug (2b43980)

  • test pars (f601362)

  • further-lm-improvments (63048eb)

  • get_lm_working_well (eba5f79)

  • basic-lm-training-currently-broken (7396b8b)

  • set_up_lm_runner (d1095af)

  • fix old test, may remove (b407aab)

  • happy with hyperpars on benchmark (836298a)

  • improve metrics (f52c7bb)

  • make toy model runner (4851dd1)

  • various-changes-toy-model-test (a61b75f)

  • Added activation store and activation gathering (a85f24d)

  • First successful run on toy models (4927145)

  • halfway-to-toy-models (feeb411)

  • Initial commit (7a94b0e)

Assets 2
Loading