Skip to content

issues Search Results · repo:EleutherAI/elk language:Python

Filter by

90 results
 (128 ms)

90 results

inEleutherAI/elk (press backspace or delete to remove)

EleutherAI/elk
Roadmaps of elk

Hello, I found elk very helpful since I am new to llm transparency. It seems elk has not been maintained for a year and what a pity. Why some method like CCS has been removed from main branch and is there ...
  • merlinarer
  • Opened 
    on Oct 12, 2024
  • #299

EleutherAI/elk
Can this code suppot free-form QA task?

I noticed that the datasets supported in the code are all multiple-choice and classification types, such as IMDB, QNLI, and BoolQ. Can the code in this repository support free-form types of tasks, such ...
  • Tan-Hexiang
  • 2
  • Opened 
    on Jul 4, 2024
  • #298

EleutherAI/elk
pip install from git doesn't work

Reproduced on my local setup and on colab !pip install git+https://github.com/EleutherAI/elk/ import elk ---- 2 import elk [/usr/local/lib/python3.10/dist-packages/elk/__init__.py](https://localhost:8080/#) ...
  • Butanium
  • Opened 
    on Jan 25, 2024
  • #297

EleutherAI/elk
Make into a `pytest.fixture` which was a TODO in `test_smoke_eval.py`

https://github.com/RohitRathore1/elk/blob/84e99a36a5050881d85f1510a2486ce46ac1f942/tests/test_smoke_eval.py#L19C1-L20C35
  • RohitRathore1
  • Opened 
    on Jan 18, 2024
  • #295

EleutherAI/elk
Evaluate LM output performance in `elk eval`

Right now only elk elicit emits an lm_eval.csv but this would also be useful to have for eval
enhancement
  • norabelrose
  • Opened 
    on Jul 20, 2023
  • #276

EleutherAI/elk
Encoder-only models do not expect `labels` argument to be passed to forward

Reproduce elicit microsoft/deberta-v2-xxlarge dataset or elicit allenai/unifiedqa-v2-t5-11b-1363200 dataset --use_encoder_states Full stack trace: ╭───────────────────── Traceback (most recent ...
  • AugustasMacijauskas
  • 2
  • Opened 
    on Jun 26, 2023
  • #262

EleutherAI/elk
Use shrinkage for (cross-)covariance estimation

We re now using the shrinkage technique from this paper in the concept-erasure repo; it makes covariance estimation robust to small sample sizes. Might make CRC-TPC, VINC, etc. work better def gaussian_shrinkage(S_hat: ...
  • norabelrose
  • 3
  • Opened 
    on Jun 22, 2023
  • #261

EleutherAI/elk
--max_examples seems to be broken

elk sweep --models huggyllama/llama-13b --datasets imdb --num_gpus 1 --net ccs is fine but elk sweep --models huggyllama/llama-13b --datasets imdb --num_gpus 1 --net ccs --max_examples 10 10 has a bug ...
  • derpyplops
  • 2
  • Opened 
    on Jun 15, 2023
  • #257

EleutherAI/elk
Cannot run README command `elk elicit microsoft/deberta-v2-xxlarge-mnli imdb`

Summary Hi there, I was attempting to set up the application based on the Quick Start guide in the README. When trying to run the elk elicit microsoft/deberta-v2-xxlarge-mnli imdb command, the program ...
  • rusheb
  • 2
  • Opened 
    on Jun 15, 2023
  • #256

EleutherAI/elk
Write smoke test for eval

Look at tests/test_smoke_elicit.py for a reference
enhancement
good first issue
  • derpyplops
  • Opened 
    on Jun 13, 2023
  • #255
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Restrict your search to the title by using the in:title qualifier.
Issue search results · GitHub