Skip to content

Commit

Permalink
chore: rename project to sghi-etl-core
Browse files Browse the repository at this point in the history
After further RnD, it's now clear it is possible to have the same
abstractions for both simpler and complex ETL workflows. Complex
workflows in this context refer to ETL workflows that support
streaming and batching. Therefore, one API specifications project is
sufficient to handle both use cases. Hence, the renaming of the project
to a more generic name to emphasize that this is a general API for
describing ETL workflows.
  • Loading branch information
kennedykori committed Mar 12, 2024
1 parent 863929b commit 584a010
Show file tree
Hide file tree
Showing 7 changed files with 38 additions and 39 deletions.
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# SGHI miniETL Core
# SGHI ETL Core

[![Checked with pyright](https://microsoft.github.io/pyright/img/pyright_badge.svg)](https://microsoft.github.io/pyright/)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/savannahghi/sghi-ml-pipeline/blob/main/LICENSE)

sghi-miniETL-core specifies the API of simple ETL workflows. That is, this
`sghi-etl-core` specifies the API of simple ETL workflows. That is, this
project defines and specifies the interfaces of the main components needed to
implement an ETL workflow.

Expand All @@ -24,6 +24,6 @@ pre-commit install

## License

[MIT License](https://github.com/savannahghi/sghi-miniETL-core/blob/main/LICENSE)
[MIT License](https://github.com/savannahghi/sghi-etl-core/blob/main/LICENSE)

Copyright (c) 2024, Savannah Informatics Global Health Institute
10 changes: 5 additions & 5 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

author = "Savannah Global Health Institute"
copyright = f"{datetime.today().year}, Savannah Global Health Institute"
project = "sghi-miniETL"
project = "sghi-etl-core"


# -----------------------------------------------------------------------------
Expand Down Expand Up @@ -60,12 +60,12 @@
("py:class", "_RDT"), # private type annotations
("py:class", "TracebackType"), # Used as type annotation. Only available when type checking
("py:class", "sghi.disposable.Disposable"), # docs aren't published yet
("py:class", "sghi.miniETL.core._PDT"), # private type annotations
("py:class", "sghi.miniETL.core._RDT"), # private type annotations
("py:class", "sghi.etl.core._PDT"), # private type annotations
("py:class", "sghi.etl.core._RDT"), # private type annotations
("py:exc", "sghi.disposable.ResourceDisposedError"), # docs aren't published yet
("py:func", "sghi.disposable.not_disposed"), # docs aren't published yet
("py:obj", "sghi.miniETL.core._PDT"), # private type annotations
("py:obj", "sghi.miniETL.core._RDT"), # private type annotations
("py:obj", "sghi.etl.core._PDT"), # private type annotations
("py:obj", "sghi.etl.core._RDT"), # private type annotations
]

templates_path = ["templates"]
Expand Down
20 changes: 10 additions & 10 deletions docs/index.rst
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
.. sghi-miniETL-core documentation master file, created by
sphinx-quickstart on Thu Jan 11 01:28:14 2024.
.. sghi-etl-core documentation master file, created by sphinx-quickstart on
Thu Jan 11 01:28:14 2024.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
.. image:: images/sghi_logo.webp
:align: center

SGHI miniETL Core
=================
SGHI ETL Core
=============

sghi-miniETL-core specifies the API of simple ETL workflows. That is, this
project defines and specifies the interfaces of the main components needed to
implement an ETL workflow.
sghi-etl-core specifies the API of simple ETL workflows. That is, this project
defines and specifies the interfaces of the main components needed to implement
an ETL workflow.

Installation
------------
Expand All @@ -20,11 +20,11 @@ We recommend using the latest version of Python. Python 3.11 and newer is
supported. We also recommend using a `virtual environment`_ in order
to isolate your project dependencies from other projects and the system.

Install the latest sghi-miniETL-core version using pip:
Install the latest sghi-etl-core version using pip:

.. code-block:: bash
pip install sghi-miniETL-core
pip install sghi-etl-core
API Reference
Expand All @@ -36,7 +36,7 @@ API Reference
:caption: API
:recursive:

sghi.miniETL
sghi.etl.core


.. _virtual environment: https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments
10 changes: 5 additions & 5 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ maintainers = [
{name = "Kennedy Kori", email = "kennedykori47@gmail.com"},
{name = "Salad Guyo", email = "saladguyo60@gmail.com"}
]
name = "sghi-miniETL-core"
name = "sghi-etl-core"
readme = "README.md"
requires-python = ">=3.11" # Support Python 3.10+.

Expand Down Expand Up @@ -71,10 +71,10 @@ test = [
]

[project.urls]
changelog = "https://github.com/savannahghi/sghi-miniETL-core/blob/develop/docs/CHANGELOG.md"
documentation = "https://github.com/savannahghi/sghi-miniETL-core/blob/develop/README.md"
homepage = "https://github.com/savannahghi/sghi-miniETL-core/"
repository = "https://github.com/savannahghi/sghi-miniETL-core.git"
changelog = "https://github.com/savannahghi/sghi-etl-core/blob/develop/docs/CHANGELOG.md"
documentation = "https://github.com/savannahghi/sghi-etl-core/blob/develop/README.md"
homepage = "https://github.com/savannahghi/sghi-etl-core/"
repository = "https://github.com/savannahghi/sghi-etl-core.git"

[tool.black]
extend-exclude = """
Expand Down
8 changes: 4 additions & 4 deletions src/sghi/miniETL/core.py → src/sghi/etl/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ class Source(Disposable, Generic[_RDT], metaclass=ABCMeta):
In a typical ETL workflow, the `Extract` phase corresponds to the ``draw``
method of this class. Consequently, a ``Source`` forms the initial step of
an SGHI ETL workflow and is thus executed first. The obtained data is then
passed to a :class:`~sghi.miniETL.core.Processor` for further processing.
passed to a :class:`~sghi.etl.core.Processor` for further processing.
.. tip::
Expand Down Expand Up @@ -76,10 +76,10 @@ class Processor(Disposable, Generic[_RDT, _PDT], metaclass=ABCMeta):
In a typical ETL workflow, the `Transform` phase is functionally equivalent
to the ``process`` method of this class. Accordingly, a ``Processor`` is
thus executed immediately after the :class:`~sghi.miniETL.core.Source`
thus executed immediately after the :class:`~sghi.etl.core.Source`
finishes in an SGHI ETL workflow. The raw data obtained from the ``Source``
is taken as input. The output of the ``Processor`` is then passed to a
:class:`~sghi.miniETL.core.Sink` for storage or transmission.
:class:`~sghi.etl.core.Sink` for storage or transmission.
.. tip::
Expand Down Expand Up @@ -118,7 +118,7 @@ class Sink(Disposable, Generic[_PDT], metaclass=ABCMeta):
"""An entity that consumes processed data.
This interface represents entities that consume processed data(the output
of a :meth:`data process operation<sghi.miniETL.core.Processor.process>`)
of a :meth:`data process operation<sghi.etl.core.Processor.process>`)
and the final step of an SGHI ETL workflow. Subclasses implementing this
interface should override the :meth:`drain` method to specify how the
processed data is consumed.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Tests for the ``sghi.miniETL.core.config`` module."""
"""Tests for the ``sghi.etl.core.config`` module."""

from __future__ import annotations

Expand All @@ -9,7 +9,7 @@
from typing_extensions import override

from sghi.disposable import not_disposed
from sghi.miniETL.core import Processor, Sink, Source
from sghi.etl.core import Processor, Sink, Source

# =============================================================================
# TESTS HELPERS
Expand Down Expand Up @@ -88,7 +88,7 @@ def dispose(self) -> None:


class TestSource(TestCase):
"""Tests for the :class:`sghi.miniETL.core.Source` interface."""
"""Tests for the :class:`sghi.etl.core.Source` interface."""

@override
def setUp(self) -> None:
Expand All @@ -104,18 +104,18 @@ def tearDown(self) -> None:
def test_invoking_source_as_a_callable_returns_expected_value(
self,
) -> None:
""":class:`~sghi.miniETL.core.Source` should return the expected
""":class:`~sghi.etl.core.Source` should return the expected
value when invoked as a callable.
In short, ensure that invoking a ``Source`` instance as a callable
delegates the actual call to :meth:`~sghi.miniETL.core.Source.draw`.
delegates the actual call to :meth:`~sghi.etl.core.Source.draw`.
""" # noqa: D202, D205

assert tuple(self._instance()) == (0, 1, 2, 3, 4)


class TestProcessor(TestCase):
"""Tests for the :class:`sghi.miniETL.core.Processor` interface."""
"""Tests for the :class:`sghi.etl.core.Processor` interface."""

@override
def setUp(self) -> None:
Expand All @@ -133,20 +133,20 @@ def tearDown(self) -> None:
def test_invoking_processor_as_a_callable_returns_expected_value(
self,
) -> None:
""":class:`~sghi.miniETL.core.Processor` should return the expected
""":class:`~sghi.etl.core.Processor` should return the expected
value when invoked as a callable.
In short, ensure that invoking a ``Processor`` instance as a callable
delegates the actual call to
:meth:`~sghi.miniETL.core.Processor.process`.
:meth:`~sghi.etl.core.Processor.process`.
""" # noqa: D202, D205

raw_values = self._source()
assert tuple(self._instance(raw_values)) == ("0", "1", "2", "3", "4")


class TestSink(TestCase):
"""Tests for the :class:`sghi.miniETL.core.Processor` interface."""
"""Tests for the :class:`sghi.etl.core.Processor` interface."""

@override
def setUp(self) -> None:
Expand All @@ -165,11 +165,11 @@ def tearDown(self) -> None:
self._instance.dispose()

def test_invoking_sink_as_a_callable_returns_expected_value(self) -> None:
""":class:`~sghi.miniETL.core.Sink` should return the expected value
""":class:`~sghi.etl.core.Sink` should return the expected value
when invoked as a callable.
In short, ensure that invoking a ``Sink`` instance as a callable
delegates the actual call to :meth:`~sghi.miniETL.core.Sink.drain`.
delegates the actual call to :meth:`~sghi.etl.core.Sink.drain`.
""" # noqa: D202, D205

self._instance(self._processor(self._source()))
Expand Down
1 change: 0 additions & 1 deletion test/sghi/mini_etl_tests/__init__.py

This file was deleted.

0 comments on commit 584a010

Please sign in to comment.