Skip to content

Commit

Permalink
simple example and whylog divider, regrouping integrations table
Browse files Browse the repository at this point in the history
  • Loading branch information
lalmei committed Apr 30, 2021
1 parent 5b44ac7 commit ba0a8dc
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 26 deletions.
72 changes: 46 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,23 +26,19 @@ This is a Python implementation of whylogs. The Java implementation can be found
If you have any questions, comments, or just want to hang out with us, please join [our Slack channel](http://join.slack.whylabs.ai/).


_____________
Latest release: 0.4.5

_____________
<img align="center" src="images/Whylabs-Dots-Light-Bg.png" width="300">
<!-- <img align="center" src="images/Whylabs-Dots-Light-Bg.png" width="300"> -->


- [Getting started](#getting-started)
- [Features](#features)
- [Data Types](#data-types)
- [Examples](#examples)
- [Integrations](#integrations)
- [Community](#community)
- [Roadmap](#roadmap)
- [Contribute](#contribute)



## Getting started<a name="getting-started" />


Expand All @@ -58,15 +54,41 @@ pip install whylogs

- Download the source code by cloning the repository or by pressing [Download ZIP](https://github.com/whylabs/whylogs-python/archive/master.zip) on this page.

- Install [poetry](https://python-poetry.org/docs/#installation) for package management
- You'll need to install poetry in order to install dependencies using the lock file in this project. Follow [their docs](https://python-poetry.org/docs/) to get it set up.

- Then running:
- Run the following comand at the root of the source code:

```
make install
make
```

## Quickly Logging Data

whylogs is easy to get up and runnings

```python
from whylogs import get_or_create_session
import pandas as pd

session = get_or_create_session()

df = pd.read_csv("path/to/file.csv")

with session.logger(dataset_name="my_dataset") as logger:

#dataframe
logger.log_dataframe(df)

#dict
logger.log({"name": 1})

#images
logger.log_images("path/to/image.png")
```

Check the examples below for visualization and other use cases

### Documentation

The [documentation](https://docs.whylabs.ai/docs/) of this package is generated automatically.
Expand All @@ -88,31 +110,29 @@ Whylogs supports both structured and unstructured data, specifically:
| --- | --- | ---|
|Structured data | Distribution, cardinality, schema, counts, missing values | [Getting started with structure data](https://github.com/whylabs/whylogs-examples/blob/mainline/python/GettingStarted.ipynb) |
| Images | exif metadata, derived pixels features, bounding boxes | [Getting started with images](https://github.com/whylabs/whylogs-examples/blob/mainline/python/Logging_Images.ipynb) |
|Video | In development | [Github Issue #214](https://github.com/whylabs/whylogs/issues/214) |
| Tensors | derived 1d features (more in developement) | [Github Issue #216](https://github.com/whylabs/whylogs/issues/216) |
| Embeddings | derived 1d features (more in developement) | |
| Text | top k values, counts, cardinality (more in developement) | [Github Issue #213](https://github.com/whylabs/whylogs/issues/213)|
| Video | In development | [Github Issue #214](https://github.com/whylabs/whylogs/issues/214) |
| Tensors | derived 1d features (more in developement) | [Github Issue #216](https://github.com/whylabs/whylogs/issues/216) |
| Text | top k values, counts, cardinality (more in developement) | [Github Issue #213](https://github.com/whylabs/whylogs/issues/213) |
| Audio | In developement | [Github Issue #212](https://github.com/whylabs/whylogs/issues/212) |


## Integrations

![current integration](images/integrations.001.png)

| Integration | Features | Notebook | Blog Post |
| --- | --- | --- | --- |
| Spark | Log and monitor any Spark dataframe | | |
| Pandas | Log and monitor any pandas dataframe | [Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/logging_example.ipynb) | [whylogs: Embrace Data Logging](https://whylabs.ai/blog/posts/whylogs-embrace-data-logging) |
| Kafka | Log and monitor Kafka topics with whylogs| [Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/Kafka.ipynb) | [Integrating whylogs into your Kafka ML Pipeline](https://whylabs.ai/blog/posts/integrating-whylogs-into-your-kafka-ml-pipeline) |
| MLflow | Enhance MLflow metrics with whylogs: | [Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/MLFlow%20Integration%20Example.ipynb) | [Streamlining data monitoring with whylogs and MLflow](https://whylabs.ai/blog/posts/on-model-lifecycle-and-monitoring) |
| Github actions | Unit test data with whylogs and github actions| [Notebook Example](https://github.com/whylabs/whylogs-examples/tree/mainline/github-actions) | |
| RAPIDS | Use whylogs in RAPIDS environment | [Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/RAPIDS%20GPU%20Integration%20Example.ipynb)| [Monitoring High-Performance Machine Learning Models with RAPIDS and whylogs](https://whylabs.ai/blog/posts/monitoring-high-performance-machine-learning-models-with-rapids-and-whylogs) |
| Java | Run whylogs in Java environment| [Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/java/demo1/src/main/java/com/whylogs/examples/WhyLogsDemo.java) | |
| Scala | Run whylogs in Scala environment| [Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/scala/src/main/scala/WhyLogsDemo.scala) | |
| Docker | Run whylogs as in Docker | | |
| AWS S3 | Store whylogs profiles in S3 | [S3 example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/S3%20example.ipynb)
| Integration | Features | Resources |
| --- | --- | --- |
| Spark | Log and monitor any Spark dataframe | |
| Pandas | Log and monitor any pandas dataframe | <ul><li>[Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/logging_example.ipynb)</li><li>[whylogs: Embrace Data Logging](https://whylabs.ai/blog/posts/whylogs-embrace-data-logging)</li></ul> |
| Kafka | Log and monitor Kafka topics with whylogs| <ul><li>[Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/Kafka.ipynb)</li><li> [Integrating whylogs into your Kafka ML Pipeline](https://whylabs.ai/blog/posts/integrating-whylogs-into-your-kafka-ml-pipeline) </li></ul>|
| MLflow | Enhance MLflow metrics with whylogs: | <ul><li>[Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/MLFlow%20Integration%20Example.ipynb)</li><li>[Streamlining data monitoring with whylogs and MLflow](https://whylabs.ai/blog/posts/on-model-lifecycle-and-monitoring)</li></ul> |
| Github actions | Unit test data with whylogs and github actions| <ul><li>[Notebook Example](https://github.com/whylabs/whylogs-examples/tree/mainline/github-actions)</li></ul> |
| RAPIDS | Use whylogs in RAPIDS environment | <ul><li>[Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/RAPIDS%20GPU%20Integration%20Example.ipynb)</li><li>[Monitoring High-Performance Machine Learning Models with RAPIDS and whylogs](https://whylabs.ai/blog/posts/monitoring-high-performance-machine-learning-models-with-rapids-and-whylogs)</li></ul> |
| Java | Run whylogs in Java environment| <ul><li>[Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/java/demo1/src/main/java/com/whylogs/examples/WhyLogsDemo.java)</li></ul> |
| Scala | Run whylogs in Scala environment| <ul><li>[Notebook Example](https://github.com/whylabs/whylogs-examples/blob/mainline/scala/src/main/scala/WhyLogsDemo.scala)</li></ul> |
| Docker | Run whylogs as in Docker | |
| AWS S3 | Store whylogs profiles in S3 | <ul><li>[S3 example](https://github.com/whylabs/whylogs-examples/blob/mainline/python/S3%20example.ipynb)</li></ul>


## Roadmap

whylogs is maintained by [WhyLabs](https://whylabs.ai).
Expand All @@ -124,7 +144,7 @@ If you have any questions, comments, or just want to hang out with us, please jo

## Contribute

For more information on contributing to whylogs, see [DEVELOPMENT.md](https://github.com/whylabs/whylogs/blob/mainline/DEVELOPMENT.md).
We welcome contributions to whylogs. Please see our [developement guide](https://github.com/whylabs/whylogs/blob/mainline/DEVELOPMENT.md) for details.



Expand Down
Binary file added images/Whylabs-Dots-Light-Bg.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit ba0a8dc

Please sign in to comment.