Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RPD-260] Add an object to handle the matcha.config.json file. #184

Merged
merged 8 commits into from
Aug 4, 2023
Merged

[RPD-260] Add an object to handle the matcha.config.json file. #184

merged 8 commits into from
Aug 4, 2023

Conversation

swells2020
Copy link
Contributor

@swells2020 swells2020 commented Aug 3, 2023 •
edited
Loading

This PR introduces a new module matcha_config.py to store the classes and services associated with the matcha.config.json file. This PR introduces an object representation of the matcha.config.json file, alongside a service class for interacting with the MatchaConfig object.

The service class implements two methods currently needed for interacting with the matcha.config.json file; read_matcha_config, for reading in data from the config JSON; and delete_matcha_config, for removing the config file.

Checklist

Please ensure you have done the following:

  • I have read the CONTRIBUTING guide.
  • I have updated the documentation if required.
  • I have added tests which cover my changes.

Type of change

Tick all those that apply:

  • Bug Fix (non-breaking change, fixing an issue)
  • New feature (non-breaking change to add functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Other (add details above)

@swells2020 swells2020 changed the title [WIP] RPD-260 [WIP] RPD-260 Add an object to handle the matcha.config.json file. Aug 3, 2023
@swells2020 swells2020 self-assigned this Aug 3, 2023
@JonoCX JonoCX changed the title [WIP] RPD-260 Add an object to handle the matcha.config.json file. [RPD-260] Add an object to handle the matcha.config.json file. Aug 3, 2023
KirsoppJ
KirsoppJ suggested changes Aug 4, 2023
View reviewed changes
Copy link
Contributor

@KirsoppJ KirsoppJ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, just a few changes mostly docstring related

src/matcha_ml/config/matcha_config.py Outdated Show resolved Hide resolved
src/matcha_ml/config/matcha_config.py Outdated Show resolved Hide resolved
src/matcha_ml/config/matcha_config.py Outdated Show resolved Hide resolved
src/matcha_ml/config/matcha_config.py Outdated Show resolved Hide resolved
src/matcha_ml/state/remote_state_manager.py Show resolved Hide resolved
tests/test_state/test_remote_state_manager.py Outdated Show resolved Hide resolved
KirsoppJ
KirsoppJ suggested changes Aug 4, 2023
View reviewed changes
src/matcha_ml/config/matcha_config.py Outdated Show resolved Hide resolved
@KirsoppJ KirsoppJ self-requested a review August 4, 2023 09:48
KirsoppJ
KirsoppJ approved these changes Aug 4, 2023
View reviewed changes
Copy link
Contributor

@KirsoppJ KirsoppJ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

JonoCX
JonoCX requested changes Aug 4, 2023
View reviewed changes
Copy link
Member

@JonoCX JonoCX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice - see comments 🦖

src/matcha_ml/config/matcha_config.py Show resolved Hide resolved
src/matcha_ml/config/matcha_config.py Outdated Show resolved Hide resolved
tests/test_config/test_matcha_config.py Outdated
for key, value in mocked_matcha_config_json_object["remote_state_bucket"].items():
properties.append(MatchaConfigComponentProperty(name=key, value=value))

return MatchaConfigComponent(name="remote_state_bucket", properties=properties)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (alternative): could be a one-liner: return MatchaConfig(mocked_matcha_config_json_object).components[0]

tests/test_config/test_matcha_config.py Outdated Show resolved Hide resolved
tests/test_state/test_remote_state_manager.py Outdated Show resolved Hide resolved
JonoCX
JonoCX approved these changes Aug 4, 2023
View reviewed changes
Copy link
Member

@JonoCX JonoCX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@swells2020 swells2020 merged commit b26b9c2 into fixes/RPD-278 Aug 4, 2023
2 checks passed
@swells2020 swells2020 deleted the RPD-260-add-matcha-config-json-object branch August 4, 2023 10:39
KirsoppJ added a commit that referenced this pull request Aug 9, 2023
@KirsoppJ @swells2020
[RPD-283] add core function for updating stack type (#185)
7d08206 
* [RPD-287] ZenML version inference for zenserver (#180)

* various changes

* change variable names to make things more clear

* shubham's comment

* test for latest

* update existing tests

* update existing tests

* shubham latest comment

* add gitflow notation branches to ci (#183)

* remove old, commented out code

* [RPD-260] Add an object to handle the `matcha.config.json` file. (#184)

* adds logic and tests for matcha config module

* updates docstrings

* adds tests and implements config object throughout matcha

* updated for pr comments

* updates docstring

* fixes ci

* updates for comments

* enum and metaclass

* enum and metaclass

* bug removal

* docstring stack_set

* docstring file exists

* fix current tests

* add a couple tests, move logic for updating to configservice

* lowercase-ify arg to enum

* american spelling

* american spelling

* chris' comment re overwriting

* friendlier api

* Runtime error if not recognised arg type fpor update()

* Runtime error if not recognised arg type fpor update()

* update provision to use new update() API

* update stack name

* tests

* tests

* tests

* quote marks

* type version, mypy

* various review comments

* test for update() function

* update tests to include quotes.

* fixing tests

* tidy tests, test error if resources provisioned

* clearer error handling

* remove unnecessary context

* update test to new error type

---------

Co-authored-by: Callum Wells <68609181+swells2020@users.noreply.github.com>
Christopher-Norman added a commit that referenced this pull request Aug 14, 2023
@Christopher-Norman
[RPD-284] Update AzureRunner to select the Terraform files correspond…
407be1c 
…ing to the stack name in the config file (#187)

* [RPD-287] ZenML version inference for zenserver (#180)

* various changes

* change variable names to make things more clear

* shubham's comment

* test for latest

* update existing tests

* update existing tests

* shubham latest comment

* add gitflow notation branches to ci (#183)

* remove old, commented out code

* [RPD-260] Add an object to handle the `matcha.config.json` file. (#184)

* adds logic and tests for matcha config module

* updates docstrings

* adds tests and implements config object throughout matcha

* updated for pr comments

* updates docstring

* fixes ci

* updates for comments

* enum and metaclass

* enum and metaclass

* bug removal

* docstring stack_set

* docstring file exists

* fix current tests

* add a couple tests, move logic for updating to configservice

* lowercase-ify arg to enum

* american spelling

* american spelling

* chris' comment re overwriting

* friendlier api

* Runtime error if not recognised arg type fpor update()

* Runtime error if not recognised arg type fpor update()

* update provision to use new update() API

* update stack name

* tests

* tests

* tests

* Move files for selection

* Clean up file

* Remove a test that no longer describes the expected functionality

* Remove a test that no longer describes the expected functionality

* Fix tests

* Stop creation of config file in local directory when running tests

* Update check to use MatchaConfig object instead of to dict

* Add get current stack name function to MatchaConfigService

* Update get stack function

* Fix test

* Remove unnecessary lower casing

* Fix tests
@JonoCX JonoCX mentioned this pull request Aug 14, 2023
Merged
7 tasks
JonoCX added a commit that referenced this pull request Aug 15, 2023
@Christopher-Norman @swells2020
@KirsoppJ @JonoCX
[Feature] LLM Stack (#191)
65a7795 
* [RPD-282] adds stack cli command (#179)

* adds tests and logic for stack and stack set cli commands

* fixes typos

* updated for comments

* addresses comments

* [RPD-285] Create LLM stack Terraform files (#182)

* Add initial LLM Terraform files

* Update Terraform Kubernetes Chroma deployment

* Update Terraform docs

* Revert default variables

* Add missing aks module

* [RPD-283] add core function for updating stack type (#185)

* [RPD-287] ZenML version inference for zenserver (#180)

* various changes

* change variable names to make things more clear

* shubham's comment

* test for latest

* update existing tests

* update existing tests

* shubham latest comment

* add gitflow notation branches to ci (#183)

* remove old, commented out code

* [RPD-260] Add an object to handle the `matcha.config.json` file. (#184)

* adds logic and tests for matcha config module

* updates docstrings

* adds tests and implements config object throughout matcha

* updated for pr comments

* updates docstring

* fixes ci

* updates for comments

* enum and metaclass

* enum and metaclass

* bug removal

* docstring stack_set

* docstring file exists

* fix current tests

* add a couple tests, move logic for updating to configservice

* lowercase-ify arg to enum

* american spelling

* american spelling

* chris' comment re overwriting

* friendlier api

* Runtime error if not recognised arg type fpor update()

* Runtime error if not recognised arg type fpor update()

* update provision to use new update() API

* update stack name

* tests

* tests

* tests

* quote marks

* type version, mypy

* various review comments

* test for update() function

* update tests to include quotes.

* fixing tests

* tidy tests, test error if resources provisioned

* clearer error handling

* remove unnecessary context

* update test to new error type

---------

Co-authored-by: Callum Wells <68609181+swells2020@users.noreply.github.com>

* [RPD-286] Add documentation on "stacks" and the new LLM stack (#189)

* Initial version of docs

* Add motivation to stacks docs

* Various docs updates based on reviews

* [RPD-284] Update AzureRunner to select the Terraform files corresponding to the stack name in the config file (#187)

* [RPD-287] ZenML version inference for zenserver (#180)

* various changes

* change variable names to make things more clear

* shubham's comment

* test for latest

* update existing tests

* update existing tests

* shubham latest comment

* add gitflow notation branches to ci (#183)

* remove old, commented out code

* [RPD-260] Add an object to handle the `matcha.config.json` file. (#184)

* adds logic and tests for matcha config module

* updates docstrings

* adds tests and implements config object throughout matcha

* updated for pr comments

* updates docstring

* fixes ci

* updates for comments

* enum and metaclass

* enum and metaclass

* bug removal

* docstring stack_set

* docstring file exists

* fix current tests

* add a couple tests, move logic for updating to configservice

* lowercase-ify arg to enum

* american spelling

* american spelling

* chris' comment re overwriting

* friendlier api

* Runtime error if not recognised arg type fpor update()

* Runtime error if not recognised arg type fpor update()

* update provision to use new update() API

* update stack name

* tests

* tests

* tests

* Move files for selection

* Clean up file

* Remove a test that no longer describes the expected functionality

* Remove a test that no longer describes the expected functionality

* Fix tests

* Stop creation of config file in local directory when running tests

* Update check to use MatchaConfig object instead of to dict

* Add get current stack name function to MatchaConfigService

* Update get stack function

* Fix test

* Remove unnecessary lower casing

* Fix tests

* updates stack handling

* Revert "updates stack handling"

This reverts commit b42b31d.

* [RPD-289] Update the Chroma Terraform within the LLM stack to use Helm #194

* [RPD-292] [BUG] Update AzureTemplate to not create redundant folders during provisioning (#195)

* splits default stack

* updates llm stack

* [RPD-303] Update stack_set docstring to include example and raises (#196)

---------

Co-authored-by: Callum Wells <68609181+swells2020@users.noreply.github.com>
Co-authored-by: KirsoppJ <40233184+KirsoppJ@users.noreply.github.com>
Co-authored-by: Jonathan Carlton <JonoCX@users.noreply.github.com>
JonoCX added a commit that referenced this pull request Aug 15, 2023
@JonoCX @KirsoppJ @swells2020
[RPD-278] Bug Fixes and Improvements (#193)
54afd3d 
* [RPD-287] ZenML version inference for zenserver (#180)

* various changes

* change variable names to make things more clear

* shubham's comment

* test for latest

* update existing tests

* update existing tests

* shubham latest comment

* add gitflow notation branches to ci (#183)

* remove old, commented out code

* [RPD-260] Add an object to handle the `matcha.config.json` file. (#184)

* adds logic and tests for matcha config module

* updates docstrings

* adds tests and implements config object throughout matcha

* updated for pr comments

* updates docstring

* fixes ci

* updates for comments

* [RPD-249] Refactor `analytics_service` to simplify tracking decorator (#181)

* refactor analytics service

* adds tests and updates for comments

* updates tests

* updates for comments

* extracts _get_state_uuid logic into private function

* removes old _get_state_uuid logic

* updates _execute_analytics_event_function

* extracts _time_event logic

* adds _post_event logic

* adds tests

* adds dataclass

* updates tests

* [RPD-236] Improved approach to check ignored folder when uploading and downloading (#186)

* [RPD-250] Move `_show_terraform_outputs()` into `provision` (#188)

* updates core and azure_runner

* adds test

* updated for comments

* updates test fixtures

* [RPD-259] Refactor `build_state_from_terraform_output` within `matcha_state.py` to use objects defined within matcha_state.py (#190)

* updates build_state_from_terraform_outputs

* updates for comments

* fixes tests

* removes print statement

* updates get_component

* updates get_component

* updates for comments

* [RPD-290] Update MatchaConfig object to not throw an error when a matching property/component is not found (#192)

---------

Co-authored-by: KirsoppJ <40233184+KirsoppJ@users.noreply.github.com>
Co-authored-by: Callum Wells <68609181+swells2020@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JonoCX JonoCX JonoCX approved these changes

@KirsoppJ KirsoppJ KirsoppJ approved these changes

Assignees

@swells2020 swells2020

Labels
improvement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@swells2020 @JonoCX @KirsoppJ