Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some important docs updates #2463

Merged
merged 49 commits into from
Feb 23, 2024
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
0c21fef
Add mention of deploying zenml in installation
htahir1 Feb 22, 2024
05b50a0
Add mention of deploying zenml in installation
htahir1 Feb 22, 2024
285e4d4
Add mention of deploying zenml in installation
htahir1 Feb 22, 2024
83f448b
ZenML Cloud
htahir1 Feb 22, 2024
7614908
More ZenML Cloud
htahir1 Feb 22, 2024
82d16ba
Added assets
htahir1 Feb 22, 2024
043d823
README
htahir1 Feb 22, 2024
c64eb8c
READ
htahir1 Feb 22, 2024
fb9f311
latest
htahir1 Feb 22, 2024
3d2e333
latest
htahir1 Feb 22, 2024
01b200e
README
htahir1 Feb 22, 2024
62f10c0
README
htahir1 Feb 22, 2024
ff24e8d
README
htahir1 Feb 22, 2024
84216a5
README
htahir1 Feb 22, 2024
e9c5a95
README
htahir1 Feb 22, 2024
0b69f5e
README
htahir1 Feb 22, 2024
2111e60
README
htahir1 Feb 22, 2024
ae3e6df
Optimised images with calibre/image-actions
github-actions[bot] Feb 22, 2024
e21cded
README
htahir1 Feb 22, 2024
f6d1d20
Merge branch 'doc/some-imp-updates' of github.com:zenml-io/zenml into…
htahir1 Feb 22, 2024
91d7f44
Update README.md
htahir1 Feb 22, 2024
a2decb9
Update docs/book/deploying-zenml/zenml-cloud/cloud-system-architectur…
htahir1 Feb 22, 2024
b8f25a0
Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md
htahir1 Feb 22, 2024
489a68c
Update docs/book/getting-started/installation.md
htahir1 Feb 22, 2024
ca6934d
Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md
htahir1 Feb 22, 2024
205cf95
Update docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md
htahir1 Feb 22, 2024
f35f54f
Update README.md
htahir1 Feb 22, 2024
edc3f0a
Merge branch 'doc/some-imp-updates' of github.com:zenml-io/zenml into…
htahir1 Feb 22, 2024
21b5ab8
Update README.md
htahir1 Feb 23, 2024
b5ea991
Update README.md
htahir1 Feb 23, 2024
6fc61c4
Update README.md
htahir1 Feb 23, 2024
c6f52fa
Update README.md
htahir1 Feb 23, 2024
1af467d
Update README.md
htahir1 Feb 23, 2024
99caf74
Update README.md
htahir1 Feb 23, 2024
869b2ae
latest
htahir1 Feb 23, 2024
d7fd32e
README again
htahir1 Feb 23, 2024
4ab7ab3
More WHy
htahir1 Feb 23, 2024
d64cdc1
Latest
htahir1 Feb 23, 2024
90613b4
Latest
htahir1 Feb 23, 2024
c7fd707
Latest
htahir1 Feb 23, 2024
4d56ae1
ZenmL Cloud
htahir1 Feb 23, 2024
481918e
Update README.md
htahir1 Feb 23, 2024
b0491dd
README again
htahir1 Feb 23, 2024
8eb35a1
Merge branch 'doc/some-imp-updates' of github.com:zenml-io/zenml into…
htahir1 Feb 23, 2024
973942d
Apply suggestions from code review
htahir1 Feb 23, 2024
6fa2bc3
latest
htahir1 Feb 23, 2024
6cef4b0
Update docs/book/getting-started/installation.md
htahir1 Feb 23, 2024
4d91af4
Merge branch 'doc/some-imp-updates' of github.com:zenml-io/zenml into…
htahir1 Feb 23, 2024
20cd28a
latest
htahir1 Feb 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 25 additions & 115 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,10 +68,11 @@
[build-url]: https://github.com/zenml-io/zenml/actions/workflows/ci.yml

<div align="center">
<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=0fcbab94-8fbe-4a38-93e8-c2348450a42e" />
<h3 align="center">Build portable, production-ready MLOps pipelines.</h3>
<p align="center">
<div align="center">
Join our <a href="https://zenml.io/slack-invite" target="_blank">
Join our <a href="https://zenml.io/slack" target="_blank">
<img width="18" src="https://cdn3.iconfinder.com/data/icons/logos-and-brands-adobe/512/306_Slack-512.png" alt="Slack"/>
<b>Slack Community</b> </a> and be part of the ZenML family.
</div>
Expand All @@ -88,7 +89,7 @@
·
<a href="https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+first+issue%22">Contribute to Open Source</a>
·
<a href="https://www.zenml.io/company#team">Meet the Team</a>
<a href="https://github.com/zenml-io/zenml-projects">Projects Showcase</a>
<br />
<br />
🎉 Version 0.55.3 is out. Check out the release notes
Expand All @@ -104,21 +105,15 @@
<details>
<summary>🏁 Table of Contents</summary>
<ol>
<li><a href="#-introduction">Introduction</a></li>
<li><a href="#-quickstart">Quickstart</a></li>
<li><a href="#🤖-introduction">Introduction</a></li>
<li><a href="#🤸-quickstart">Quickstart</a></li>
<li>
<a href="#-create-your-own-mlops-platform">Create your own MLOps Platform</a>
<ul>
<li><a href="##-1-deploy-zenml">Deploy ZenML</a></li>
<li><a href="#-2-deploy-stack-components">Deploy Stack Components</a></li>
<li><a href="#-3-create-a-pipeline">Create a Pipeline</a></li>
<li><a href="#-4-start-the-dashboard">Start the Dashboard</a></li>
</ul>
<a href="#🖼️-learning">Learning</a>
</li>
<li><a href="#-roadmap">Roadmap</a></li>
<li><a href="#-contributing-and-community">Contributing and Community</a></li>
<li><a href="#-getting-help">Getting Help</a></li>
<li><a href="#-license">License</a></li>
<li><a href="#🗺-roadmap">Roadmap</a></li>
<li><a href="#🙌-contributing-and-community">Contributing and Community</a></li>
<li><a href="#🆘-getting-help">Getting Help</a></li>
<li><a href="#📜-license">License</a></li>
</ol>
</details>

Expand All @@ -141,17 +136,14 @@ experimentation while writing code that is production-ready from the get-go.
- 🛫 ZenML enables MLOps infrastructure experts to define, deploy, and manage
sophisticated production environments that are easy to use for colleagues.

![The long journey from experimentation to production.](/docs/book/.gitbook/assets/intro-zenml-overview.png)

ZenML provides a user-friendly syntax designed for ML workflows, compatible with
any cloud or tool. It enables centralized pipeline management, enabling
developers to write code once and effortlessly deploy it to various
infrastructures.

<div align="center">
<img src="docs/book/.gitbook/assets/overview.gif">
<img width="60%" src="/docs/book/.gitbook/assets/zenml-hero.png" alt="ZenML Hero"/>
</div>

ZenML aims to optimize your entire machine learning lifecycle - you'll be 80% faster to production, generate 3x more models, and massively reduce your cloud spend.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe instead of "aims to", something more active?

  • ZenML optimizes your entire...
  • ZenML is all about optimizing your...
  • etc

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also feels like maybe these very precise numbers need links to user case studies etc. without links it feels a bit arbitrary? (Not sure how others would read it, though).


![The long journey from experimentation to production.](/docs/book/.gitbook/assets/zenml-why.png)

# 🤸 Quickstart

[Install ZenML](https://docs.zenml.io/getting-started/installation) via
Expand All @@ -167,113 +159,31 @@ Take a tour with the guided quickstart by running:
zenml go
```

# 🖼️ Create your own MLOps Platform

ZenML allows you to create and manage your own MLOps platform using
best-in-class open-source and cloud-based technologies. Here is an example of
how you could set this up for your team:

## 🔋 1. Deploy ZenML

For full functionality ZenML should be deployed on the cloud to
enable collaborative features as the central MLOps interface for teams.

![ZenML Architecture Diagram.](docs/book/.gitbook/assets/Scenario3.png)

Currently, there are two main options to deploy ZenML:

- **ZenML Cloud**: With [ZenML Cloud](https://docs.zenml.io/deploying-zenml/zenml-cloud),
you can utilize a control plane to create ZenML servers, also known as tenants.
These tenants are managed and maintained by ZenML's dedicated team, alleviating
the burden of server management from your end.

- **Self-hosted deployment**: Alternatively, you have the flexibility to [deploy
ZenML on your own self-hosted environment](https://docs.zenml.io/deploying-zenml/zenml-self-hosted).
This can be achieved through various methods, including using our CLI, Docker,
Helm, or HuggingFace Spaces.

## 👨‍🍳 2. Deploy Stack Components

ZenML boasts a ton of [integrations](https://zenml.io/integrations) into
popular MLOps tools. The [ZenML Stack](https://docs.zenml.io/user-guide/starter-guide/understand-stacks)
concept ensures that these tools work nicely together, therefore bringing
structure and standardization into the MLOps workflow.

Deploying and configuring this is super easy with ZenML. For **AWS**, this might
look a bit like this

```bash
# Deploy and register an orchestrator and an artifact store
zenml orchestrator deploy kubernetes_orchestrator --flavor kubernetes --cloud aws
zenml artifact-store deploy s3_artifact_store --flavor s3

# Register this combination of components as a stack
zenml stack register production_stack --orchestrator kubernetes_orchestrator --artifact-store s3_artifact_store --set # Register your production environment
```

When you run a pipeline with this stack set, it will be running on your deployed
Kubernetes cluster.
# 🖼️ Learning

You can also [deploy your own tooling manually](https://docs.zenml.io/stacks-and-components/stack-deployment).
The best way to learn about ZenML is the [docs](https://docs.zenml.io) - with the best path being through the [Starter Guide](https://docs.zenml.io/user-guide/starter-guide).
htahir1 marked this conversation as resolved.
Show resolved Hide resolved

## 🏇 3. Create a Pipeline
For inspiration, here are some other examples & use-cases:
htahir1 marked this conversation as resolved.
Show resolved Hide resolved

Here's an example of a hello world ZenML pipeline in code:

```python
# run.py
from zenml import pipeline, step


@step
def step_1() -> str:
"""Returns the `world` substring."""
return "world"


@step
def step_2(input_one: str, input_two: str) -> None:
"""Combines the two strings at its input and prints them."""
combined_str = input_one + ' ' + input_two
print(combined_str)


@pipeline
def my_pipeline():
output_step_one = step_1()
step_2(input_one="hello", input_two=output_step_one)


if __name__ == "__main__":
my_pipeline()
```

```bash
python run.py
```

## 👭 4. Start the Dashboard

Open up the ZenML dashboard using this command.

```bash
zenml show
```
1. [E2E Batch Inference](examples/e2e/): Feature engineering, training, and inference pipelines for tabular machine learning.
2. [Basic NLP with BERT](examples/e2e_nlp/): Feature engineering, training, and inference focused on NLP.
3. [LLM RAG Pipeline with Langchain and OpenAI](https://github.com/zenml-io/zenml-projects/tree/main/llm-agents): Using Langchain to create a simple RAG pipeline.
4. [Huggingface Model to Sagemaker Endpoint](https://github.com/zenml-io/zenml-projects/tree/main/huggingface-sagemaker): Automated MLOps on Amazon Sagemaker and HuggingFace.

# 🗺 Roadmap

ZenML is being built in public. The [roadmap](https://zenml.io/roadmap) is a
regularly updated source of truth for the ZenML community to understand where
the product is going in the short, medium, and long term.

ZenML is managed by a [core team](https://zenml.io/company#CompanyTeam) of
ZenML is managed by a [core team](https://zenml.io/company) of
developers that are responsible for making key decisions and incorporating
feedback from the community. The team oversees feedback via various channels,
and you can directly influence the roadmap as follows:

- Vote on your most wanted feature on our [Discussion
board](https://zenml.io/discussion).
- Start a thread in our [Slack channel](https://zenml.io/slack-invite).
- Start a thread in our [Slack channel](https://zenml.io/slack).
- [Create an issue](https://github.com/zenml-io/zenml/issues/new/choose) on our
GitHub repo.

Expand All @@ -291,7 +201,7 @@ Guide](CONTRIBUTING.md) for all relevant details.
The first point of call should
be [our Slack group](https://zenml.io/slack-invite/).
Ask your questions about bugs or specific use cases, and someone from
the [core team](https://zenml.io/company#CompanyTeam) will respond.
the [core team](https://zenml.io/company) will respond.
Or, if you
prefer, [open an issue](https://github.com/zenml-io/zenml/issues/new/choose) on
our GitHub repo.
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/book/.gitbook/assets/zenml-hero.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/book/.gitbook/assets/zenml-why.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Different variations of the ZenML Cloud architecture depending on y
# System Architecture

{% hint style="info" %}
If you would like to evaluate ZenML Cloud, please [book a demo](https://zenml.io/book-a-demo) and we can help you decide what scenario suits you best!
Should you be interested in assessing ZenML Cloud, you can create a [free account](https://cloud.zenml.io), which defaults to a [Scenario 1](#scenario-1-full-saas) deployment. To upgrade to different scenarios, please [reach out to us](mailto:cloud@zenml.io).
htahir1 marked this conversation as resolved.
Show resolved Hide resolved
{% endhint %}

Machine learning often involves data that is sensitive and thus data security is a fundamental requirement. The ZenML Cloud is flexible enough to meet your security requirements, from easy installations to completely airgapped deployments.
Expand All @@ -16,7 +16,7 @@ The ZenML Cloud consists of the following moving pieces:
* **Single Sign-On (SSO)**: The ZenML Cloud API is integrated with [Auth0](https://auth0.com/) as an SSO provider to manage user authentication and authorization. Users can log in to the ZenML Cloud dashboard using their social media accounts or their corporate credentials.
* **Secrets Store**: All secrets and credentials required to access customer infrastructure services are stored in a secure secrets store. The ZenML Cloud API has access to these secrets and uses them to access customer infrastructure services on behalf of the ZenML Cloud. The secrets store can be hosted either by the ZenML Cloud or by the customer.
* **ML Metadata Store**: This is where all ZenML metadata is stored, including ML metadata such as tracking and versioning information about pipelines and models.
* **ZenML Cloud Agent**: This service is optionally deployed customer-side, and interacts with customer MLOps stack components on behalf of the remote ZenML Cloud control plane.
* **ZenML Cloud Agent (Optional)**: This service is optionally deployed customer-side, and interacts with customer MLOps stack components on behalf of the remote ZenML Cloud control plane.

The above five interact with other MLOps stack components, secrets, and data in varying scenarios described below.

Expand Down
42 changes: 29 additions & 13 deletions docs/book/deploying-zenml/zenml-cloud/zenml-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,28 +4,44 @@ description: Your one-stop MLOps control plane.

# ☁ ZenML Cloud

ZenML Cloud is a Software-as-a-Service (SaaS) platform that extends the capabilities of the open-source ZenML product. It provides you with a centralized control plane to effortlessly launch and manage their ZenML server instances. The core foundation of ZenML Cloud remains the powerful open-source offering, but with ZenML Cloud, you gain access to a host of additional features designed to streamline the machine learning workflow.
One of the most straightforward paths to start with a deployed ZenML server is through [ZenML Cloud](https://zenml.io/cloud). ZenML Cloud eliminates the need for you to dedicate time and resources to deploy and manage the ZenML server, allowing you to focus primarily on your MLOps workflows.
htahir1 marked this conversation as resolved.
Show resolved Hide resolved

<div data-full-width="true">

<figure><img src="../../.gitbook/assets/cloud_architecture_scenario_1.png" alt=""><figcaption><p>An architectural overview of the ZenML Cloud</p></figcaption></figure>

<figure><img src="../../.gitbook/assets/zenml-cloud-tenant-overview.png" alt=""><figcaption><p>ZenML Cloud comes equipped with a powerful cloud-only dashboard</p></figcaption></figure>
</div>

{% hint style="info" %}
Should you be interested in assessing ZenML Cloud, you can simply create a [free account](https://cloud.zenml.io). Learn more about ZenML Cloud on the [ZenML Website](https://zenml.io/cloud).
htahir1 marked this conversation as resolved.
Show resolved Hide resolved
{% endhint %}

## Key features

ZenML Cloud brings convenience to your machine learning workflows by allowing you to deploy a managed instance of ZenML servers with a single click. This means you can set up and manage your machine learning pipelines effortlessly without the hassle of dealing with infrastructure complexities. We take care of all the necessary upgrades and backups, ensuring that your system remains up-to-date and resilient while you focus on your core MLOps tasks. As a valued cloud customer, you'll also benefit from priority support, ensuring that you receive the assistance you need to make the most of our platform.
ZenML Cloud is a Software-as-a-Service (SaaS) platform that enhances the functionalities of the open-source ZenML product. It equips you with a centralized interface to seamlessly launch and manage ZenML server instances. While it remains rooted in the robust open-source offering, ZenML Cloud offers added features designed to optimize your machine learning workflow.
htahir1 marked this conversation as resolved.
Show resolved Hide resolved

Data security and privacy are of utmost importance at ZenML Cloud. You can connect your infrastructure safely and securely and we track only metadata through an encrypted connection, ensuring that your sensitive information remains confidential. Our platform offers seamless integration with your cloud services through service connectors, making it easy to connect and utilize various cloud resources without compromising data security. Additionally, we store your secrets in a safe and secure isolated environment, providing an extra layer of protection for your valuable data.
### Managed ZenML Server (Multi-tenancy)

As a ZenML Cloud user, you also gain early access to a powerful control plane that centralizes user management and streamlines your workflows. With centralized management of users, handling access and permissions becomes efficient and organized. Moreover, ZenML Cloud users enjoy exclusive access to a range of cloud-only features, providing you with a competitive edge and the opportunity to stay ahead in the rapidly evolving field of machine learning.
ZenML Cloud simplifies your machine learning workflows, enabling you to deploy a managed instance of ZenML servers with just one click. This eradicates the need to handle infrastructure complexities, making the set-up and management of your machine learning pipelines a breeze. We handle all pertinent system updates and backups, thus ensuring your system stays current and robust, allowing you to zero in on your essential MLOps tasks. As a ZenML Cloud user, you'll also have priority support, giving you the necessary aid to fully utilize the platform.

{% hint style="info" %}
ZenML Cloud is currently in the beta phase, offering users the opportunity to host a managed ZenML instance and gain early access to the exciting new features mentioned above. Beta users will receive priority access to the enhanced functionalities and dedicated support to ensure a smooth onboarding experience.
{% endhint %}
### Maximum data security

At ZenML Cloud, your data security and privacy are our top priority. The platform enables a secure connection to your infrastructure, tracking only metadata via an encrypted connection to maintain the confidentiality of your sensitive information. ZenML Cloud integrates smoothly with your cloud services via service connectors, allowing a straightforward connection with various cloud resources without sacrificing data security. We hold your confidential information in a secure and isolated environment, offering an extra degree of protection. If desired, you can even [supply your own secret store](cloud-system-architecture.md#scenario-2-hybrid-saas-with-customer-secret-store-managed-by-zenml).

Click [here](cloud-system-architecture.md) to understand about the ZenML Cloud system architecture.

### Role-based access control and permissions

Utilizing ZenML Cloud provides you with early access to a robust control plane that unifies user management and optimizes your workflows. Efficiently manage access and permissions through centralized user administration. Create fine-grained permissions for resources such as stacks, pipelines, models, etc.

### A brand-new, modern MLOps experience

![Walkthrough of ZenML Model Control Plane](../../.gitbook/assets/mcp_walkthrough.gif)

We have built the ZenML Cloud from the ground-up. With ZenML Cloud, you get access to a new dashboard, with a better experience. The new dashboard features more functionality such as the [Model Control Plane](../../user-guide/starter-guide/track-ml-models.md) and [Artifact Control Plane](../../user-guide/starter-guide/manage-artifacts.md).

### Triggers, CI/CD, Reports and more

## Coming soon...
Additionally, ZenML Cloud users get exclusive access to an array of cloud-specific features, such as triggering pipelines directly through the dashboard, integrating with your code repository CI/CD system, generating usage reports and more.

<table data-card-size="large" data-view="cards" data-full-width="true"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:purple;"><strong>CI/CD/CT with Triggers</strong></mark></td><td>Continuous Integration, Continuous Deployment, and Continuous Training are all integrated into your repository.</td><td><a href="https://zenml.io/cloud-features/continuous-integration-and-delivery-ci-cd">https://zenml.io/cloud-features/continuous-integration-and-delivery-ci-cd</a></td></tr><tr><td><mark style="color:purple;"><strong>ML Model Control Plane</strong></mark></td><td>All your models are conveniently gathered in one place. Up-to-date information about your training, deployments, and endpoints all within one view.</td><td><a href="https://zenml.io/cloud-features/ml-models-control-plane">https://zenml.io/cloud-features/ml-models-control-plane</a></td></tr><tr><td><mark style="color:purple;"><strong>Managed MLOps infrastructure</strong></mark></td><td>ZenML Cloud also includes hosted instances of support stack components like MLflow.</td><td><a href="https://zenml.io/cloud-features/managed-ml-ops-infrastructure">https://zenml.io/cloud-features/managed-ml-ops-infrastructure</a></td></tr><tr><td><mark style="color:purple;"><strong>Tenants and Workspaces</strong></mark></td><td>Create multiple ZenML tenants and workspaces within tenants to subdivide your team's efforts neatly.</td><td><a href="https://zenml.io/cloud-features/tenants-and-workspaces">https://zenml.io/cloud-features/tenants-and-workspaces</a></td></tr><tr><td><mark style="color:purple;"><strong>User Management</strong></mark></td><td>User permission management and project-specific configurations</td><td><a href="https://zenml.io/cloud-features/user-management">https://zenml.io/cloud-features/user-management</a></td></tr><tr><td><mark style="color:purple;"><strong>Role Based Access Control</strong></mark></td><td>Manage permissions and roles at an organizational level</td><td><a href="https://zenml.io/cloud-features/role-based-access-control">https://zenml.io/cloud-features/role-based-access-control</a></td></tr></tbody></table>
Learn more about ZenML Cloud on the [ZenML Website](https://zenml.io/cloud).

<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>
<figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>
Loading