Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BIT-484] Server/validator improvements (TextCausalLMNext) #852

Merged
merged 42 commits into from
Jul 26, 2022
Merged

[BIT-484] Server/validator improvements (TextCausalLMNext) #852

merged 42 commits into from
Jul 26, 2022

Conversation

opentaco
Copy link
Contributor

@opentaco opentaco commented Jul 24, 2022 •
edited
Loading

BIT-484 Server/validator improvements (TextCausalLMNext)

Various synapse improvements, highlights include:

  1. Critical tokenizer fixes like setting server tokenizer padding_side.
  2. Support for facebook/opt-* models and other non-fast tokenizer models.
  3. Validator checks for NaNs in neuron_stats.
  4. Gradual EMA from zero for neuron_stats used for weighting (to improve mean estimation).
  5. Added normal validation loss to TextCausalLMNext.

Tested branch successfully for core_server and core_validator on CUDA, including running facebook/opt-13b.

@opentaco
Fix topk not implemented on cpu for float16
7bab953
@opentaco
Unforce remote_train when cuda and float16
aec9a30
@opentaco
Ensure probability computations done in float32 for improved precision
b248cac
@opentaco
Set tokenizer.padding_side = "left" in core_server
4c7436c 
Generative default expects most recent token on right-hand side with padding on left. huggingface/transformers#10552
@opentaco
Run model for TextCausalLMNext always
2e47182 
Do not reuse model outputs from TextCausalLM, since the padding side differs.
@opentaco
Gradually increase weight fields from zero via EMA
1ddcd26 
New axons will gradually increase in weighting as the number of successfully responded queries grows. This ensures that sufficient observations are averaged before weighting to address potentially noisy validation measures.
@opentaco
Add vocab to tokenizer if it does not have attr
d9c60f1
@opentaco
Add vocab to tokenizer if it does not have attr
0512641
@opentaco
Define vocab_len as real tokenizer vocabulary length
27658df
@opentaco
Merge remote-tracking branch 'origin/BIT-484-7-TextCausalLMNext-dev' …
29e51e3 
…into BIT-484-7-TextCausalLMNext-dev

# Conflicts:
#	bittensor/_neuron/text/core_server/nucleus_impl.py
@opentaco
Define vocab_len as real tokenizer vocabulary length
d11b6cf
@opentaco
Try use_fast=False at tokenizer load if fast fails
aa44ed3
@opentaco
Add epsilon for log in loss computation
a964bde
@opentaco
Sort validation table according to TextCausalLMNext
27a1b4c 
More models, like OPT, are supported by TextCausalLMNext than TextCausalLM that requires fast tokenizers. Validation table sorted according to more populated synapse provides better view.
@opentaco
Avoid EMA of nan values in core_validator neuron_stats
d049163
@opentaco
Replace nan/inf losses with large loss in core_validator
4101ff5
@opentaco
Swap validator table columns TextCausalLM <-> TextCausalLMNext
c62dbad
@opentaco
Ensure softmax computation done in float32 for improved precision
e7d44c0
@opentaco
Simplify remapping_token to include untensorized offset_mappings
355c328
@opentaco
Simplify encode_forward_causallm in core_server
97e7987
@opentaco
Remove left padding from batch probabilities for logit translation
d479000
@opentaco
Remove left padding from batch tokens for logit translation
9685a65
@opentaco
Return internal model_output in encode_forward_causallm
4da7f67
@opentaco
Reuse model_output in encode_forward_causallmnext in core_server
f52b2f8
@opentaco
Move tokens to core_server device
57c7ced
@opentaco
Set std_sequence_len according to tokens_std in logit translation
85b302f
@opentaco
Check that model_output is not None before overwrite in axon
3318cfd 
Otherwise some synapses like TextSeq2Seq with model_output=None will overwrite previous (potentially) non-None model_output.
@opentaco
Revert model_output check in axon
090bc36
@opentaco
Log server loss and translated loss for TextCausalLM in core_server
a6df74d
@opentaco
Update target_phrases type
986799b
@coveralls
Copy link

coveralls commented Jul 24, 2022 •
edited
Loading

Pull Request Test Coverage Report for Build 0febfdd3-ca3e-4d28-b483-65e1b15a6c30

  • 42 of 50 (84.0%) changed or added relevant lines in 1 file are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage decreased (-0.2%) to 64.806%
Changes Missing Coverage Covered Lines Changed/Added Lines %
bittensor/utils/tokenizer_utils.py 42 50 84.0%
Files with Coverage Reduction New Missed Lines %
bittensor/utils/tokenizer_utils.py 1 85.81%
Totals Coverage Status
Change from base Build 5ab03dc9-731c-4622-bbe5-c614b3b857bc: -0.2%
Covered Lines: 3924
Relevant Lines: 6055
💛 - Coveralls

Eugene-hu
Eugene-hu approved these changes Jul 26, 2022
View reviewed changes
Copy link
Contributor

@Eugene-hu Eugene-hu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Amazing work!

bittensor/_neuron/text/core_server/nucleus_impl.py Show resolved Hide resolved
bittensor/_neuron/text/core_server/nucleus_impl.py Show resolved Hide resolved
bittensor/_neuron/text/core_validator/__init__.py Show resolved Hide resolved
bittensor/_neuron/text/core_validator/__init__.py Show resolved Hide resolved
@opentaco opentaco merged commit 23b004f into Synapse Jul 26, 2022
@ifrit98 ifrit98 deleted the BIT-484-7-TextCausalLMNext-dev branch May 24, 2023 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Eugene-hu Eugene-hu Eugene-hu approved these changes

@isabella618033 isabella618033 Awaiting requested review from isabella618033

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@opentaco @coveralls @Eugene-hu