Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Projects] Sync SDK class with API methods + structure change #596

Merged
merged 29 commits into from
Dec 17, 2020

Conversation

Hedingber
Copy link
Contributor

@Hedingber Hedingber commented Dec 16, 2020

  • So far the SDK class was completely disconnected from the API, (confusingly) even doing .save() would cause the project to be saved to a file, there was no useful way to actually back up the projects to the DB. (Note the projects were getting created in the DB when you created a resource of a project (function,run,artifact etc..) but it was only the name without all the project data) - This PR changed that, from now on .save() would save to the DB (in addition to saving to file) + all of the API methods (httpdb) are accepting and returning MlrunProject instances, so you can get a project from the db by doing:
from mlrun import get_run_db
get_run_db().get_project(<name>)
  • Since [Projects] Enable projects syncing with Nuclio #542 anyways does non-backwards compatible changes to the projects API, I used the option and did a restructure to the project object so that it will have metadata, spec and status like our other resources. Except than different output file format this should be completely transparent to the user, the code "knows" how to load from the old structure file, and there are "proxy attributes" so if your code is doing project.name (while it should do project.metadata.name) it will work and won't explode (though there is a deprecation plan for these proxy attributes)
    So a project file looked something like:
name: churn-project
functions:
- name: clean-data
  spec:
    kind: job
    metadata:
      name: clean-data
      tag: ''
      project: churn-project
    spec:
      command: ''
      args: []
      image: mlrun/ml-models
      env: []
      default_handler: ''
    verbose: false
- url: hub://describe
  name: describe
workflows:
- name: workflow-name
  path: /Users/hediingber/iguazio/misc-workspace/mlrun/mlrun/tests/projects/workflow.py
artifacts:
- key: raw-data
  kind: ''
  iter: 0
  tree: latest
  target_path: https://raw.githubusercontent.com/mlrun/demos/master/customer-churn-prediction/WA_Fn-UseC_-Telco-Customer-Churn.csv
  db_key: raw-data
artifact_path: /User/mlrun-demos/customer-churn-prediction/data

will now look something like:

kind: project
metadata:
  name: last-project
spec:
  functions:
  - name: clean-data
    spec:
      kind: job
      metadata:
        name: clean-data
        tag: ''
        project: last-project
      spec:
        command: ''
        args: []
        image: mlrun/ml-models
        env: []
        default_handler: ''
      verbose: false
  - url: hub://describe
    name: describe
  workflows:
  - name: workflow-name
    path: /Users/hediingber/iguazio/misc-workspace/mlrun/mlrun/tests/projects/workflow.py
  artifacts:
  - key: raw-data
    kind: ''
    iter: 0
    tree: latest
    target_path: https://raw.githubusercontent.com/mlrun/demos/master/customer-churn-prediction/WA_Fn-UseC_-Telco-Customer-Churn.csv
    db_key: raw-data
  artifact_path: /User/mlrun-demos/customer-churn-prediction/data
  source: ''
  context: ./project
  subpath: ''
  origin_url: ''
  tag: ''

Should be Merged/released together with mlrun/ui#322

@Hedingber Hedingber merged commit 3f8cfa3 into mlrun:development Dec 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant