Convert examples to conform to best practices

[AlexejPenner]

Describe changes

I have updated some of our examples to conform to our best practices - ✔️ is done - 🍔 is not done and should be done in the next few days - airflow currently does not work with our best practices structure.

✔️ add_your_own
✔️ airflow_orchestration
✔️ cloud_secrets_manager
✔️ evidently_drift_detection
✔️ facets_visualize_statistics
✔️ feast_feature_store
✔️ github_actions_orchestration
✔️ : huggingface
✔️ kubeflow_pipelines_orchestration
✔️ lightgbm
✔️ mlflow_deployment
✔️ mlflow_tracking
✔️ neural_prophet
✔️ pytorch
✔️ quickstart
✔️ scipy
🌵 seldon_deployment - done - Need to be tested with AWS cluster where seldon is installed
✔️ slack_alert
✔️ step_operator_remote_training
✔️ vertex_ai_orchestration
✔️ wandb_tracking
✔️ whylogs_data_profiling
✔️ xgboost

https://docs.zenml.io/links-and-resources/resources/best-practices

Besides the obvious many changes to the example structure there have been a few changes in the core repo. The pipeline-run code has moved out of the cli into the pipelines module to be callable from code without invoking subprocess calls or cli-runners.

On top of that the source utils have been adjusted to make sure that zenml pipeline run ... -c config.yaml works for our recommended repo structure. Special attention should be payed to the changes in src/zenml/utils/source_utils.py during code review (:eyes: @schustmi 😉 )

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• If my change requires a change to docs, I have updated the documentation accordingly.
• If I have added an integration, I have updated the integrations table.
• I have added tests to cover my changes.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[htahir1]

The fact that ✅ is the emoji for not done is the biggest troll today! :D

[AlexejPenner]

The fact that white_check_mark is the emoji for not done is the biggest troll today! :D

Changed it to 🍔 to motivate volunteers

[fa9r]

I can do the alerter one later and will do the PR review once all examples are done.

Also, could you please add a link to the best practices section somewhere in your PR description?

[strickvl]

@fa9r https://docs.zenml.io/links-and-resources/resources/best-practices is the best practices link, but you probably would do better just to look at the model of one of the complete ones. I was going to write up instructions on this process on Friday but didn't get around to it. Maybe overkill given how many we have left and that this is a one-off process. But lmk if you have any questions. I didn't find it super intuitive to do the first one.

[fa9r]

slack example should also be done now

[github-actions]

@htahir1, @fa9r
1.25 (one and twenty-five hundredths) business days have passed since the review started. Pretty please review the PR.

[fa9r]

We're getting close, really looking forward to finally have this merged! 😄

[AlexejPenner]

Just some tests missing now :)

[safoinme]

Only Seldon's example is left for the test I think. tried to make sure it's right but still would be better to run it if you guys can do that with the AWS Stack @AlexejPenner @fa9r

[safoinme]

Somehow not all examples are really following the same exact format! there are examples where each step or pipeline is inside a separated folder and some are gathered together, should it be unified @AlexejPenner @strickvl ?

[strickvl]

@safoinme like which examples?

[safoinme]

• examples where all steps are gathered together: kubeflow_pipeline_orchestration, PyTorch, quickstart, slack_alerter
• examples where each step is in a separated folder: mlflow_tracking, neural_prophet, Scipy, seldon_deployment, step_operator, Vertex_ai, wandb, why_logs, xgboost,
  @strickvl

[stefannica]

I know I'm late to this party, but I just had to go through the process of refactoring one of my examples to conform to these best practices and I have to object to the python module hierarchy recommendation. Do we really need to have a separate python sub-module for every single pipeline and step ? I find it unnecessary and difficult to work with.

A better/simpler hierarchy would be e.g.:

example 
  /pipelines   
    __init__.py
    training.py
    inference.py
  /steps
    __init__.py
    importer.py
    splitter.py
    deployer.py
...

[schustmi]

Suggested change

# module_name = os.path.splitext(os.path.basename(file_path))[0]

[schustmi]

Can you maybe add a short (or long) comment here that explains in detail why the module needs to be deleted, as that is a very strange operation if someone comes access this code.

[fa9r]

I know I'm late to this party, but I just had to go through the process of refactoring one of my examples to conform to these best practices and I have to object to the python module hierarchy recommendation. Do we really need to have a separate python sub-module for every single pipeline and step ? I find it unnecessary and difficult to work with.

A better/simpler hierarchy would be e.g.:

example 
  /pipelines   
    __init__.py
    training.py
    inference.py
  /steps
    __init__.py
    importer.py
    splitter.py
    deployer.py
...

I agree 100%. I did all my refactorings like this, which are the "inconsistencies" between the different examples we talked about 😄 For now, I think we agreed that both structures are acceptable since this design choice doesn't affect testing.