Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Grouped Query Attention #199

Draft
wants to merge 322 commits into
base: main
Choose a base branch
from
Draft

Grouped Query Attention #199

wants to merge 322 commits into from

Conversation

dereklai1
Copy link
Contributor

@dereklai1 dereklai1 commented Jul 6, 2024 •
edited
Loading

PR for GQA final year project.

This branch is re-based on feature/bert so it contains many duplicate changes from #184.

Changes:

dereklai1 and others added 30 commits March 15, 2024 17:26
@dereklai1
fixed isqrt linting error
1eba479
@dereklai1
removed unused params from tb
d069332
@dereklai1
pipelined isqrt module and integrated into group norm 2d
609963a
@dereklai1
added new rms norm extension layers, hardware not passing tb
36886f0
@dereklai1
fixed rms norm implementation and added tests for hw
524073a
@dereklai1
added rms norm to software stack
3183671
@dereklai1
refactored rms_norm and test_emit_verilog_norm test
69d1b11
@dereklai1
commented out rms_norm circular dependency issue not fixed
dbad6f2
@dereklai1
normtb works for rms
d042424
@dereklai1
removed online layer norm
5334943
@dereklai1
moved inverse sqrt C++ fixed model
aeb1bfb
@dereklai1
fixed groupnorm net in test_emit_verilog_norm
6fe7f42
@dereklai1
refactored test_emit_verilog_norm and readjusted test times
31a60c1
@dereklai1
docstring
3c199b4
@dereklai1
add memoryfile and top for vivado synth
ef48c93
vivado script
f67e60b
fixed top level params & ports
bcd2382
@J-u1i0
Batch norm 2d working
6b4d93b
@J-u1i0
Merge branch 'layernorm_sqrt' of github.com:dereklai1/mase into layer…
8f7a796 
…norm_sqrt
@dereklai1
added comments
b1f00e1
added u250 constraints file and updated tcl
1ae1eaf
@dereklai1
stripped xdc file
8a956a7
@dereklai1
docstrings
82de30c
@J-u1i0
Batch norm hook up?
c8e71d9
@J-u1i0
Merge branch 'layernorm_sqrt' of github.com:dereklai1/mase into layer…
a5185f2 
…norm_sqrt
@dereklai1
integrated batchnorm into norm.sv all tests passing
0a18c29
@dereklai1
pipelined batchnorm module
528ba4c
@dereklai1
Merge branch 'main' into group_17_norm
6030acc
@dereklai1
added docs for normalization layers
503f72d
@dereklai1
fixed bugged channel selector
9c0471d
Aaron-Zhao123 and others added 29 commits May 24, 2024 15:41
@Aaron-Zhao123
fix reg_slicer related stuff when calling python src/mase_components/…
9ea719b 
…linear/test/test_lint_linear.py
@Aaron-Zhao123
Merge remote-tracking branch 'refs/remotes/origin/feature/bert' into …
0c6f92e 
…feature/bert
@Aaron-Zhao123
now passed python3 test/passes/graph/transforms/verilog/test_emit_ver…
7e08de4 
…ilog_bert.py for real
@pgimenes
nas bert
308aca3
@pgimenes
Merge branch 'feature/bert' of https://github.com/DeepWok/mase into f…
b8c2ff3 
…eature/bert
@pgimenes
[NAS-BERT] layerwise search space working
0ac62fd
@pgimenes
[NAS-BERT] include num heads/hidden dim in search space, plot pareto
17d4a24
@dereklai1
patch combinatorial bugs from matmul
a2026d7
@dereklai1
Changes to get evaluation of gqa
b469e1b
@pgimenes
compatibility changes for questasim
28b5b4e
@pgimenes
comparator fix to pass larger config
911cfe7
@pgimenes
fix mount for beholder and register old gelu removal
340f9b7
@pgimenes
make simulator choice explicit
8158db1
@pgimenes
initialize llama, mistral
568a822
@dereklai1
Merge branch 'main' into gqa
a5f8fac
@dereklai1
Merge remote-tracking branch 'mase/feature/bert' into gqa
d206691
@dereklai1
reformat pow2 test & fix streaming merge
5281a07
@dereklai1
working file
777c63f
@dereklai1
commited local changes
52614e0
@dereklai1
eval changes
628a06e
@dereklai1
slight changes
4e0165e
llama2-150m model
e925742
@dereklai1
llama2 uptraining for language modelling working
24bc89f
changes to get questa starting up
a7f8c58
questa changes
bd3b4e4
@dereklai1
GQA broken
dc25b86
@dereklai1
fix regslice
a0deb0f
@dereklai1
GQA verilog emit
6ff1edf
@dereklai1
removed useless gqa modules
749b1c4
@jianyicheng jianyicheng mentioned this pull request Jul 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@dereklai1 @J-u1i0 @jianyicheng @btarifi10 @pgimenes @Aaron-Zhao123