Merge dd2cc56 into d5f5196

.github/workflows/docs.yaml

@@ -0,0 +1,24 @@
+on: # todo
+  pull_request:
+    branches:
+      - "main"
+
+permissions:
+  contents: write
+
+jobs:
+  docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - run: pipx install poetry
+      - uses: actions/setup-python@v4
+        with:
+          python-version: "3.10"
+          cache: "poetry"
+      - run: poetry install
+
+      - run: poetry run pydoc-markdown
+
+      - run: poetry run mkdocs gh-deploy --force
+        working-directory: docs/build

.github/workflows/lint.yaml

@@ -8,7 +8,7 @@ jobs:
       - uses: actions/checkout@v3
       - uses: psf/black@stable
         with:
-          version: "23.1.0"
+          version: "23.3.0"
 
   isort:
     runs-on: ubuntu-latest

.github/workflows/release.yaml

@@ -1,11 +1,11 @@
 name: "Release"
 
 on:
-  push:
-    tags: [ "*.*.*" ]
+  release:
+    types: [ published ]
 
 concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
+  group: ${{ github.workflow }}
   cancel-in-progress: true
 
 jobs:

.gitignore

@@ -11,3 +11,6 @@ htmlcov/
 .env
 
 state.json
+coinmarket.json
+
+docs/build

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/psf/black
-    rev: 23.1.0
+    rev: 23.3.0
     hooks:
       - id: black
         language_version: python3.10

Makefile

@@ -6,3 +6,6 @@ test:
 
 lines:
 	@poetry run pygount --format=summary --suffix=py hnhm/
+
+lines-all:
+	@poetry run pygount --format=summary --suffix=py hnhm/ tests/

README.md

@@ -1,15 +1,15 @@
-# <img src="assets/logo.png" height="40px">
+# <img src="docs/assets/logo.png" height="40px">
 [![codecov](https://codecov.io/gh/marchinho11/hnhm/branch/main/graph/badge.svg?token=PFB1111T2D)](https://codecov.io/gh/marchinho11/hnhm)
 [![Release](https://github.com/marchinho11/hnhm/actions/workflows/release.yaml/badge.svg?branch=main&event=push)](https://github.com/marchinho11/hnhm/actions/workflows/release.yaml)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Downloads](https://static.pepy.tech/personalized-badge/hnhm?period=week&units=international_system&left_color=black&right_color=brightgreen&left_text=PyPi%20/%20week)](https://pepy.tech/project/hnhm)
+[![Downloads](https://static.pepy.tech/personalized-badge/hnhm?period=total&units=international_system&left_color=black&right_color=brightgreen&left_text=PyPi%20/%20total)](https://pepy.tech/project/hnhm)
+![Visitors](https://api.visitorbadge.io/api/combined?path=marchinho11%2Fhnhm&label=Visitors&labelColor=%23000000&countColor=%2347c21a&style=flat&labelStyle=none)
 
 **hNhM**(highly Normalized hybrid Model) – data modeling methodology based on Anchor modeling and Data Vault. Implementation of this package is based on report **"How we implemented our data storage model — highly Normalized hybrid Model"** by Evgeny Ermakov and Nikolai Grebenshchikov. 
 [1) Yandex Report, habr.com](https://habr.com/ru/company/yandex/blog/557140/). [2) SmartData Conference, youtube.com](https://youtu.be/2fPqDvHsd0w)
-
 * [Basic hNhM concepts](#basic-hnhm-concepts)
 * [Quick Start](#quick-start)
-* [DWH example](#dwh-example)
-* [Guide](#guide)
 
 ### Basic hNhM concepts
 **Logical level**
@@ -34,9 +34,10 @@ Install `hnhm` library
 pip install hnhm
 ```
 
-Create a directory with the name `dwh` and put the `__init__.py` file there with the following contents:
+Create a directory with the name `dwh` and put the `__hnhm__.py` file there with the following contents:
+
 ```python
-# dwh/__init__.py
+# dwh/__hnhm__.py
 from hnhm import (
     Layout,
     LayoutType,
@@ -46,8 +47,8 @@ from hnhm import (
     HnHm,
     HnhmEntity,
     HnhmRegistry,
-    FileStorage,
-    PostgresSqlalchemySql
+    FileState,
+    PostgresPsycopgSql
 )
 
 
@@ -63,12 +64,13 @@ class User(HnhmEntity):
 
     __keys__ = [user_id]
 
-
-__registry__ = HnhmRegistry(
+sql=PostgresPsycopgSql(database="hnhm", user="postgres")
+
+registry = HnhmRegistry(
     entities=[User()],
     hnhm=HnHm(
-        storage=FileStorage("state.json"),
-        sql=PostgresSqlalchemySql(database="hnhm", user="postgres"),
+        state=FileState("state.json"),
+        sql=sql,
     ),
 )
 ```
@@ -79,7 +81,7 @@ $ hnhm apply dwh
 
 Plan:
 
-+ entity 'user'
++ entity 'HNHM.user'
   + view 'user'
   + hub 'user'
   + attribute 'age'
@@ -107,8 +109,3 @@ view: entity__user
  │ + _loaded_at      │
  └───────────────────┘
 ```
-
-## DWH example
-Full DWH example including Entities, Links and Flows can be found in the [`dwh/`](dwh/) directory.
-
-## Guide

docs/assets/color_scheme.css

@@ -0,0 +1,6 @@
+[data-md-color-scheme="slate"] {
+  --md-primary-fg-color:        #2e303e; /* Panel */
+  --md-default-bg-color:        #2e303e; /* BG */
+  --md-typeset-a-color:         #8e91aa;
+  --md-accent-fg-color:         #658eda
+}

docs/index.md

@@ -0,0 +1,5 @@
+# hNhM book
+![Visits](https://api.visitorbadge.io/api/combined?path=marchinho11%2Fhnhm%2Fdocs&label=Visits&labelColor=%23000000&countColor=%2347c21a&style=flat&labelStyle=none)
+
+**hNhM**(highly Normalized hybrid Model) – data modeling methodology based on Anchor modeling and Data Vault. Implementation of this package is based on report **"How we implemented our data storage model — highly Normalized hybrid Model"** by Evgeny Ermakov and Nikolai Grebenshchikov. 
+[1) Yandex Report, habr.com](https://habr.com/ru/company/yandex/blog/557140/). [2) SmartData Conference, youtube.com](https://youtu.be/2fPqDvHsd0w)

docs/tutorial/init-hnhm.md

@@ -0,0 +1,15 @@
+# Init hNhM
+Create the `__hnhm__.py` file with the following contents:
+```python
+# __hnhm__.py
+from hnhm import PostgresPsycopgSql
+
+sql = PostgresPsycopgSql(
+    database="coinmarket",
+    user="hnhm",
+    password="123", # replace it with an environment variable in the future :)
+    port=5433,
+)
+```
+
+This `sql` object will help us to generate and execute SQL queries in the database.

docs/tutorial/intro.md

@@ -0,0 +1 @@
+# Intro

docs/tutorial/raw-data.md

@@ -0,0 +1,61 @@
+# Raw Data
+Create the `etl__raw.py` file. It does the following:
+- Fetch the latest currencies
+- Create the table in the database
+- Insert data
+
+```python
+# etl__raw.py
+import os
+
+import requests
+from psycopg2.extras import Json
+
+from tutorial.__hnhm__ import sql
+
+COINMARKET_TOKEN = os.environ["COINMARKET_TOKEN"]
+
+
+def get_listings() -> list[dict]:
+    r = requests.get(
+        "https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest",
+        headers={"X-CMC_PRO_API_KEY": COINMARKET_TOKEN},
+    )
+    assert r.status_code == 200
+
+    data = r.json()
+
+    return [
+        {"timestamp": data["status"]["timestamp"], "listing": listing}
+        for listing in data["data"]
+    ]
+
+
+def save_listings(listings: list[dict]):
+    sql.execute(
+        """
+        CREATE TABLE IF NOT EXISTS raw_listings(
+            id SERIAL PRIMARY KEY,
+            data JSONB
+        )
+    """
+    )
+
+    sql.execute_many(
+        """
+        INSERT INTO
+            raw_listings(data)
+        VALUES
+            (%s)
+    """,
+        ((Json(listing),) for listing in listings),
+    )
+
+
+if __name__ == "__main__":
+    listings = get_listings()
+    save_listings(listings)
+    print(f"Successfully saved: {len(listings)} listings!")
+
+```
+

docs/tutorial/setup.md

@@ -0,0 +1,83 @@
+# Setup
+## Setup CoinMarket account
+[Create](https://coinmarketcap.com/api/) free account to work with the CoinMarket API.
+
+## Setup Python
+* Create a directory for the guide:
+   ```shell
+   mkdir hnhm_guide
+   cd hnhm_guide
+   ```
+* Install Python with version `>= 3.10`.
+* Create virtualenv (personally, I prefer [poetry](https://python-poetry.org/)):
+   ```shell
+   python3.10 -m virtualenv venv
+   source ./venv/bin/activate
+   ```
+* Create the file `requirements.txt`:
+   ```text
+   # requirements.txt
+   requests==2.31.0
+   hnhm
+   ```
+* Install packages:
+   ```shell
+   pip install -r requirements.txt
+   ```
+
+File layout at the end of current step:
+```
+.
+├── requirements.txt
+└── venv
+```
+
+## Setup Postgres
+Let's set up Postgres database. You could use the following `docker-compose.yml` file to run Postgres:
+
+```yaml
+# docker-compose.yml 
+version: "3.9"
+
+volumes:
+  postgres_data: { }
+
+services:
+  postgres:
+    image: postgres:15
+    volumes:
+      - "postgres_data:/var/lib/postgresql/data"
+    ports:
+      - "5433:5432"
+    environment:
+      POSTGRES_DB: coinmarket
+      POSTGRES_USER: hnhm
+      POSTGRES_PASSWORD: 123
+```
+
+Run Postgres with the following command:
+```shell
+docker-compose up -d
+
+# [+] Running 3/3
+#  ✔ Network hnhm_guide_default         Created                                                                                                                                      0.1s 
+#  ✔ Volume "hnhm_guide_postgres_data"  Created                                                                                                                                      0.0s 
+#  ✔ Container hnhm_guide-postgres-1    Started
+```
+
+Now you should have access to the `coinmarket` database:
+```shell
+psql -h localhost -p 5433 -U hnhm coinmarket
+# Password for user hnhm: 123
+
+~# \dt
+# Did not find any relations.
+```
+
+File layout at the end of current step:
+```
+.
+├── docker-compose.yml
+├── requirements.txt
+└── venv
+```