v2.7.2 #363
Merged
v2.7.2 #363
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Hot fix/redirect wandb
…ery (#304) * Retry creation of challenge query multiple times
Organic Scoring Implementation Changes: - This implementation is based on the Generic Organic Scoring framework introduced [here](macrocosm-os/organic-scoring#1). - Organic scoring runs in a separate `asyncio` task alongside current benchmarking tasks. - Organic queries are received via an open validator axon and stored in the organic queue. - For each organic or synthetic query, a reference answer is generated by the LLM. - Rewards and penalties are calculated based on the `relevance` metric for both organic and synthetic queries, which is defined as the cosine similarity between sentence embeddings of the reference and completions. - Augmented LMSys-Chat is used for synthetic queries. - Logging includes elapsed time between steps inside the organic loop, organic queue length, and other default logs used by benchmarking tasks, except prompts and completions, which are excluded from logging into W&B. - Validator queries 5 random miners from the network to stream back completions for organic queries (defined in config as `neuron.organic_sample_size`). - Reward step for organic or synthetic queue is triggered every 15 seconds and scaled down to 2 seconds if the organic queue is growing (defined in config as `neuron.organic_trigger`, `neuron.organic_trigger_frequency`, and `neuron.organic_trigger_frequency_min`). Process Workflow: 1. **Trigger Check**: Upon triggering the rewarding process, the system checks if the organic queue is empty. If the queue is empty, synthetic datasets (defined in `organic_scoring/synth_dataset_base.py`) are used to bootstrap the organic scoring mechanism. Otherwise, samples from the organic queue are utilized. 2. **Data Processing**: The sampled data is concurrently passed to the `_query_miners` and `_generate_reference` methods. 3. **Reward Generation**: After receiving responses from miners and any reference data, the information is processed by the `_generate_rewards` method. 4. **Weight Setting**: The generated rewards are then applied through the `_set_weights` method. 5. **Logging**: Finally, the results can be logged using the `_log_results` method, along with all relevant data provided as arguments, and default time elapsed on each step of rewarding process.
* Hotfix undefined HumanAgent.challenge_time * Set begin_conversation to True for organics
Fix Wikipedia broken sections
Restart when an error is encountered in the get_block function. Errors when making substrate calls usually result in the validator failing quietly, often requiring a manual restart. This PR is intended to catch errors originating from calls to the Bittensor package, raise them as BittensorError, and then restart.
- Enable organic scoring weight setting. - Fix bittensor WASM errors by switching to another bittensor branch.
Fix 'MockPipeline' object has no attribute 'generate' errors when using --neuron.model_id mock.
Fix AttributeError (no attribute ‘isdigit’) for Wikipedia Summary and Date.
…-uids Log organic miner UIDs to wandb
Merge main into staging (#329)
Changes: - New multi-choice benchmarking task; - Refactor changes (.env config-based, decoupled parts of the code); - Poetry setup; - Only 5 tasks are included: QA, DateQA, Summary, MultiChoice, Organic.
Add hotkey signature to the wandb run for multi-choice verification
bkb2135
approved these changes
Sep 3, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Patch v2.7.2 changes