Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some important docs updates #2463

Merged
merged 49 commits into from
Feb 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
0c21fef
Add mention of deploying zenml in installation
htahir1 Feb 22, 2024
05b50a0
Add mention of deploying zenml in installation
htahir1 Feb 22, 2024
285e4d4
Add mention of deploying zenml in installation
htahir1 Feb 22, 2024
83f448b
ZenML Cloud
htahir1 Feb 22, 2024
7614908
More ZenML Cloud
htahir1 Feb 22, 2024
82d16ba
Added assets
htahir1 Feb 22, 2024
043d823
README
htahir1 Feb 22, 2024
c64eb8c
READ
htahir1 Feb 22, 2024
fb9f311
latest
htahir1 Feb 22, 2024
3d2e333
latest
htahir1 Feb 22, 2024
01b200e
README
htahir1 Feb 22, 2024
62f10c0
README
htahir1 Feb 22, 2024
ff24e8d
README
htahir1 Feb 22, 2024
84216a5
README
htahir1 Feb 22, 2024
e9c5a95
README
htahir1 Feb 22, 2024
0b69f5e
README
htahir1 Feb 22, 2024
2111e60
README
htahir1 Feb 22, 2024
ae3e6df
Optimised images with calibre/image-actions
github-actions[bot] Feb 22, 2024
e21cded
README
htahir1 Feb 22, 2024
f6d1d20
Merge branch 'doc/some-imp-updates' of github.com:zenml-io/zenml into…
htahir1 Feb 22, 2024
91d7f44
Update README.md
htahir1 Feb 22, 2024
a2decb9
Update docs/book/deploying-zenml/zenml-cloud/cloud-system-architectur…
htahir1 Feb 22, 2024
b8f25a0
Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md
htahir1 Feb 22, 2024
489a68c
Update docs/book/getting-started/installation.md
htahir1 Feb 22, 2024
ca6934d
Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md
htahir1 Feb 22, 2024
205cf95
Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md
htahir1 Feb 22, 2024
f35f54f
Update README.md
htahir1 Feb 22, 2024
edc3f0a
Merge branch 'doc/some-imp-updates' of github.com:zenml-io/zenml into…
htahir1 Feb 22, 2024
21b5ab8
Update README.md
htahir1 Feb 23, 2024
b5ea991
Update README.md
htahir1 Feb 23, 2024
6fc61c4
Update README.md
htahir1 Feb 23, 2024
c6f52fa
Update README.md
htahir1 Feb 23, 2024
1af467d
Update README.md
htahir1 Feb 23, 2024
99caf74
Update README.md
htahir1 Feb 23, 2024
869b2ae
latest
htahir1 Feb 23, 2024
d7fd32e
README again
htahir1 Feb 23, 2024
4ab7ab3
More WHy
htahir1 Feb 23, 2024
d64cdc1
Latest
htahir1 Feb 23, 2024
90613b4
Latest
htahir1 Feb 23, 2024
c7fd707
Latest
htahir1 Feb 23, 2024
4d56ae1
ZenmL Cloud
htahir1 Feb 23, 2024
481918e
Update README.md
htahir1 Feb 23, 2024
b0491dd
README again
htahir1 Feb 23, 2024
8eb35a1
Merge branch 'doc/some-imp-updates' of github.com:zenml-io/zenml into…
htahir1 Feb 23, 2024
973942d
Apply suggestions from code review
htahir1 Feb 23, 2024
6fa2bc3
latest
htahir1 Feb 23, 2024
6cef4b0
Update docs/book/getting-started/installation.md
htahir1 Feb 23, 2024
4d91af4
Merge branch 'doc/some-imp-updates' of github.com:zenml-io/zenml into…
htahir1 Feb 23, 2024
20cd28a
latest
htahir1 Feb 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 65 additions & 102 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,11 @@
[build-url]: https://github.com/zenml-io/zenml/actions/workflows/ci.yml

<div align="center">
<h3 align="center">Build portable, production-ready MLOps pipelines.</h3>
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=0fcbab94-8fbe-4a38-93e8-c2348450a42e" />
<h3 align="center">Create an MLOps workflow for your entire team.</h3>
<p align="center">
<div align="center">
Join our <a href="https://zenml.io/slack-invite" target="_blank">
Join our <a href="https://zenml.io/slack" target="_blank">
<img width="18" src="https://cdn3.iconfinder.com/data/icons/logos-and-brands-adobe/512/306_Slack-512.png" alt="Slack"/>
<b>Slack Community</b> </a> and be part of the ZenML family.
</div>
Expand All @@ -82,13 +83,13 @@
·
<a href="https://github.com/zenml-io/zenml/issues">Report Bug</a>
·
<a href="https://zenml.io/discussion">Vote New Features</a>
<a href="https://zenml.io/cloud">Sign up for Cloud</a>
·
<a href="https://www.zenml.io/blog">Read Blog</a>
·
<a href="https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+first+issue%22">Contribute to Open Source</a>
·
<a href="https://www.zenml.io/company#team">Meet the Team</a>
<a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
<br />
<br />
🎉 Version 0.55.3 is out. Check out the release notes
Expand All @@ -104,21 +105,15 @@
<details>
<summary>🏁 Table of Contents</summary>
<ol>
<li><a href="#-introduction">Introduction</a></li>
<li><a href="#-quickstart">Quickstart</a></li>
<li><a href="#🤖-introduction">Introduction</a></li>
<li><a href="#🤸-quickstart">Quickstart</a></li>
<li>
<a href="#-create-your-own-mlops-platform">Create your own MLOps Platform</a>
<ul>
<li><a href="##-1-deploy-zenml">Deploy ZenML</a></li>
<li><a href="#-2-deploy-stack-components">Deploy Stack Components</a></li>
<li><a href="#-3-create-a-pipeline">Create a Pipeline</a></li>
<li><a href="#-4-start-the-dashboard">Start the Dashboard</a></li>
</ul>
<a href="#🖼️-learning">Learning</a>
</li>
<li><a href="#-roadmap">Roadmap</a></li>
<li><a href="#-contributing-and-community">Contributing and Community</a></li>
<li><a href="#-getting-help">Getting Help</a></li>
<li><a href="#-license">License</a></li>
<li><a href="#🗺-roadmap">Roadmap</a></li>
<li><a href="#🙌-contributing-and-community">Contributing and Community</a></li>
<li><a href="#🆘-getting-help">Getting Help</a></li>
<li><a href="#📜-license">License</a></li>
</ol>
</details>

Expand All @@ -141,17 +136,46 @@ experimentation while writing code that is production-ready from the get-go.
- 🛫 ZenML enables MLOps infrastructure experts to define, deploy, and manage
sophisticated production environments that are easy to use for colleagues.

![The long journey from experimentation to production.](/docs/book/.gitbook/assets/intro-zenml-overview.png)

ZenML provides a user-friendly syntax designed for ML workflows, compatible with
any cloud or tool. It enables centralized pipeline management, enabling
developers to write code once and effortlessly deploy it to various
infrastructures.

<div align="center">
<img src="docs/book/.gitbook/assets/overview.gif">
<img width="60%" src="/docs/book/.gitbook/assets/zenml-hero.png" alt="ZenML Hero"/>
</div>

# 🛠️ Why ZenML?

![Walkthrough of ZenML Model Control Plane (Dashboard available only on ZenML Cloud)](/docs/book/.gitbook/assets/mcp_walkthrough.gif)

ZenML offers a systematic approach to structuring your machine learning codebase for a seamless transition to production. It's an ideal solution for teams grappling with establishing an internal standard for coordinating ML operations. ZenML provides not just a tool, but a workflow strategy that guides you in integrating all your tools and infrastructure effectively.

Use ZenML if:

- You need to easily automate ML workflows on services like an Airflow cluster or AWS Sagemaker Pipelines.
- Your ML tasks require repeatability and reproducibility.
- Automating and standardizing ML workflows across your team is a challenge.
- Your team integrates multiple tools with no central platform.
- You'd like a single place to track data, code, configuration, and models along with your cloud artifact storage.
- Collaboration and hand-overs between multiple teams is difficult.

# ☄️ What makes ZenML different?

![Before and after ZenML](/docs/book/.gitbook/assets/zenml-why.png)

ZenML marries the capabilities of a classic pipeline tool like [Airflow](https://airflow.apache.org/) and a metadata tracking service like [MLflow](https://mlflow.org/). Furthermore, both these types of tools can seamlessly co-exist with ZenML, providing a comprehensive, end-to-end ML experience.

It excels at:

- Enabling creation of simple, pythonic [ML pipelines](https://docs.zenml.io/user-guide/starter-guide/create-an-ml-pipeline) that function locally and on any [orchestration backend](https://docs.zenml.io/user-guide/production-guide/cloud-orchestration).
- Automating versioning of [data](https://docs.zenml.io/user-guide/starter-guide/manage-artifacts) and [models](https://docs.zenml.io/user-guide/starter-guide/track-ml-models) on [remote artifact storage like S3](https://docs.zenml.io/user-guide/production-guide/remote-storage).
- Abstracting infrastructure and run configuration from code through a [simple YAML config](https://docs.zenml.io/user-guide/advanced-guide/pipelining-features/configure-steps-pipelines).
- Logging complex [metadata](https://docs.zenml.io/user-guide/advanced-guide/data-management/logging-metadata) for models and artifacts.
- Automatically containerizing and deploying your workflows to the cloud, connected to your [code repository](https://docs.zenml.io/user-guide/production-guide/connect-code-repository).
- Connecting your [secret store](https://docs.zenml.io/user-guide/advanced-guide/secret-management) to your ML workflows.

However, ZenML doesn't:

- Automatically create visuals and track experiments: It [integrates with experiment trackers](https://docs.zenml.io/stacks-and-components/component-guide/experiment-trackers) that specialize in this task.
- Package and deploy models: ZenML catalogs models and metadata, streamlining model deployment. Refer to [ZenML model deployers](https://docs.zenml.io/stacks-and-components/component-guide/model-deployers) for more information.
- Handle distributed computation: While ZenML pipelines scale vertically with ease, it [works with tools like Spark](https://docs.zenml.io/stacks-and-components/component-guide/step-operators/spark-kubernetes) for intricate distributed workflows.

# 🤸 Quickstart

[Install ZenML](https://docs.zenml.io/getting-started/installation) via
Expand All @@ -167,22 +191,18 @@ Take a tour with the guided quickstart by running:
zenml go
```

# 🖼️ Create your own MLOps Platform

ZenML allows you to create and manage your own MLOps platform using
best-in-class open-source and cloud-based technologies. Here is an example of
how you could set this up for your team:

## 🔋 1. Deploy ZenML
# 🔋 Deploy ZenML

For full functionality ZenML should be deployed on the cloud to
enable collaborative features as the central MLOps interface for teams.

![ZenML Architecture Diagram.](docs/book/.gitbook/assets/Scenario3.png)
<div align="center">
<img width="60%" src="docs/book/.gitbook/assets/Scenario3.png" alt="ZenML Architecture Diagram."/>
</div>

Currently, there are two main options to deploy ZenML:

- **ZenML Cloud**: With [ZenML Cloud](https://docs.zenml.io/deploying-zenml/zenml-cloud),
- **ZenML Cloud**: With [ZenML Cloud](cloud.zenml.io/?utm_source=readme&utm_medium=referral_link&utm_campaign=cloud_promotion&utm_content=signup_link),
you can utilize a control plane to create ZenML servers, also known as tenants.
These tenants are managed and maintained by ZenML's dedicated team, alleviating
the burden of server management from your end.
Expand All @@ -192,88 +212,31 @@ ZenML on your own self-hosted environment](https://docs.zenml.io/deploying-zenml
This can be achieved through various methods, including using our CLI, Docker,
Helm, or HuggingFace Spaces.

## 👨‍🍳 2. Deploy Stack Components

ZenML boasts a ton of [integrations](https://zenml.io/integrations) into
popular MLOps tools. The [ZenML Stack](https://docs.zenml.io/user-guide/starter-guide/understand-stacks)
concept ensures that these tools work nicely together, therefore bringing
structure and standardization into the MLOps workflow.
# 🖼️ Learning

Deploying and configuring this is super easy with ZenML. For **AWS**, this might
look a bit like this
The best way to learn about ZenML is the [docs](https://docs.zenml.io). We recommend beginning with the [Starter Guide](https://docs.zenml.io/user-guide/starter-guide) to get up and running quickly.

```bash
# Deploy and register an orchestrator and an artifact store
zenml orchestrator deploy kubernetes_orchestrator --flavor kubernetes --cloud aws
zenml artifact-store deploy s3_artifact_store --flavor s3

# Register this combination of components as a stack
zenml stack register production_stack --orchestrator kubernetes_orchestrator --artifact-store s3_artifact_store --set # Register your production environment
```
For inspiration, here are some other examples and use cases:

When you run a pipeline with this stack set, it will be running on your deployed
Kubernetes cluster.

You can also [deploy your own tooling manually](https://docs.zenml.io/stacks-and-components/stack-deployment).

## 🏇 3. Create a Pipeline

Here's an example of a hello world ZenML pipeline in code:

```python
# run.py
from zenml import pipeline, step


@step
def step_1() -> str:
"""Returns the `world` substring."""
return "world"


@step
def step_2(input_one: str, input_two: str) -> None:
"""Combines the two strings at its input and prints them."""
combined_str = input_one + ' ' + input_two
print(combined_str)


@pipeline
def my_pipeline():
output_step_one = step_1()
step_2(input_one="hello", input_two=output_step_one)


if __name__ == "__main__":
my_pipeline()
```

```bash
python run.py
```

## 👭 4. Start the Dashboard

Open up the ZenML dashboard using this command.

```bash
zenml show
```
1. [E2E Batch Inference](examples/e2e/): Feature engineering, training, and inference pipelines for tabular machine learning.
2. [Basic NLP with BERT](examples/e2e_nlp/): Feature engineering, training, and inference focused on NLP.
3. [LLM RAG Pipeline with Langchain and OpenAI](https://github.com/zenml-io/zenml-projects/tree/main/llm-agents): Using Langchain to create a simple RAG pipeline.
4. [Huggingface Model to Sagemaker Endpoint](https://github.com/zenml-io/zenml-projects/tree/main/huggingface-sagemaker): Automated MLOps on Amazon Sagemaker and HuggingFace.

# 🗺 Roadmap

ZenML is being built in public. The [roadmap](https://zenml.io/roadmap) is a
regularly updated source of truth for the ZenML community to understand where
the product is going in the short, medium, and long term.

ZenML is managed by a [core team](https://zenml.io/company#CompanyTeam) of
ZenML is managed by a [core team](https://zenml.io/company) of
developers that are responsible for making key decisions and incorporating
feedback from the community. The team oversees feedback via various channels,
and you can directly influence the roadmap as follows:

- Vote on your most wanted feature on our [Discussion
board](https://zenml.io/discussion).
- Start a thread in our [Slack channel](https://zenml.io/slack-invite).
- Start a thread in our [Slack channel](https://zenml.io/slack).
- [Create an issue](https://github.com/zenml-io/zenml/issues/new/choose) on our
GitHub repo.

Expand All @@ -291,12 +254,12 @@ Guide](CONTRIBUTING.md) for all relevant details.
The first point of call should
be [our Slack group](https://zenml.io/slack-invite/).
Ask your questions about bugs or specific use cases, and someone from
the [core team](https://zenml.io/company#CompanyTeam) will respond.
the [core team](https://zenml.io/company) will respond.
Or, if you
prefer, [open an issue](https://github.com/zenml-io/zenml/issues/new/choose) on
our GitHub repo.

# Vulnerability affecting `zenml<0.67` (CVE-2024-25723)
# Vulnerability affecting `zenml<0.46.7` (CVE-2024-25723)

We have identified a critical security vulnerability in ZenML versions prior to
0.46.7. This vulnerability potentially allows unauthorized users to take
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/book/.gitbook/assets/zenml-hero.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/book/.gitbook/assets/zenml-why.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Different variations of the ZenML Cloud architecture depending on y
# System Architecture

{% hint style="info" %}
If you would like to evaluate ZenML Cloud, please [book a demo](https://zenml.io/book-a-demo) and we can help you decide what scenario suits you best!
If you're interested in assessing ZenML Cloud, you can create a [free account](https://cloud.zenml.io/?utm_source=docs&utm_medium=referral_link&utm_campaign=cloud_promotion&utm_content=signup_link), which defaults to a [Scenario 1](#scenario-1-full-saas) deployment. To upgrade to different scenarios, please [reach out to us](mailto:cloud@zenml.io).
{% endhint %}

Machine learning often involves data that is sensitive and thus data security is a fundamental requirement. The ZenML Cloud is flexible enough to meet your security requirements, from easy installations to completely airgapped deployments.
Expand All @@ -16,7 +16,7 @@ The ZenML Cloud consists of the following moving pieces:
* **Single Sign-On (SSO)**: The ZenML Cloud API is integrated with [Auth0](https://auth0.com/) as an SSO provider to manage user authentication and authorization. Users can log in to the ZenML Cloud dashboard using their social media accounts or their corporate credentials.
* **Secrets Store**: All secrets and credentials required to access customer infrastructure services are stored in a secure secrets store. The ZenML Cloud API has access to these secrets and uses them to access customer infrastructure services on behalf of the ZenML Cloud. The secrets store can be hosted either by the ZenML Cloud or by the customer.
* **ML Metadata Store**: This is where all ZenML metadata is stored, including ML metadata such as tracking and versioning information about pipelines and models.
* **ZenML Cloud Agent**: This service is optionally deployed customer-side, and interacts with customer MLOps stack components on behalf of the remote ZenML Cloud control plane.
* **ZenML Cloud Agent (Optional)**: This service is optionally deployed customer-side, and interacts with customer MLOps stack components on behalf of the remote ZenML Cloud control plane.

The above five interact with other MLOps stack components, secrets, and data in varying scenarios described below.

Expand Down Expand Up @@ -78,6 +78,6 @@ Here is a concrete example of how this works:

In this scenario, all services, data, and secrets are deployed on the customer cloud. This is the opposite of Scenario 1, and is meant for customers who require completely airgapped deployments, for the tightest security standards.

Are you interested in the ZenML Cloud? While in beta, we're looking for early adopters to talk to! [Sign up](https://cloud.zenml.io) and get access to Scenario 1. with a free 30 day trial now!
Are you interested in the ZenML Cloud? While in beta, we're looking for early adopters to talk to! [Sign up](https://cloud.zenml.io/?utm_source=docs&utm_medium=referral_link&utm_campaign=cloud_promotion&utm_content=signup_link) and get access to Scenario 1. with a free 30 day trial now!

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>
Loading
Loading