Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Representation based alignment and merge #199

Draft
wants to merge 13 commits into
base: wip-git-rebasin
Choose a base branch
from
Draft

Representation based alignment and merge #199

wants to merge 13 commits into from

Conversation

shamanez
Copy link
Member

@shamanez shamanez commented Mar 18, 2024 •
edited by metric-space 

python dump_out.py gpt2 -o dump_output --dump-type hidden-state -d metric-space/experiment_med -s 2 -c question  -u part1
python dump_out.py gpt2 -o dump_output --dump-type activation -d metric-space/experiment_med -s 2 -c question  -u part1 --layer-hooks transformer.h.10.mlp.act,transformer.h.1.mlp.act

shamanez
shamanez commented Mar 18, 2024
View reviewed changes
examples/zipit.yml
weight: 1.0
- model: garage-bAInd/Platypus2-13B
parameters:
density: 0.5
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are also some ZipIt related parameters like Alpha and Beta.

mergekit/_data/architectures/gpt2.json Outdated
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, will these modifications help us compute intermediate representations?

mergekit/_data/mappings/a_b.json Outdated
"start_architectures": ["A"],
"destination_architectures": ["B"],
"weights_mapping":{
"a" : "d",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For ZipIt should we add something like intra & inter feature mapping?

@shamanez shamanez changed the base branch from main to wip-git-rebasin March 18, 2024 18:48
@metric-space metric-space changed the title Wip zipit Representation based alignment and merge Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@shamanez @metric-space