Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DOP-16175] Add versionadded and vesionchanged directives #282

Merged
merged 1 commit into from
May 27, 2024
Merged

[DOP-16175] Add versionadded and vesionchanged directives #282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/hooks/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
Hooks
=====

.. versionadded:: 0.6.0

.. toctree::
:maxdepth: 1
:caption: Hooks
Expand Down
4 changes: 4 additions & 0 deletions docs/plugins.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
Plugins
=======

.. versionadded:: 0.6.0

What are plugins?
-----------------

Expand Down Expand Up @@ -82,6 +84,8 @@ like :ref:`hook-decorator`, it will be executed during this import.
How to enable/disable plugins?
------------------------------

.. versionadded:: 0.7.0

Disable/enable all plugins
~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
6 changes: 6 additions & 0 deletions onetl/base/base_db_connection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,9 @@ def read_source_as_df(
) -> DataFrame:
"""
Reads the source to dataframe. |support_hooks|

.. versionchanged:: 0.9.0
Renamed ``read_df`` → ``read_source_as_df``
"""

@abstractmethod
Expand All @@ -142,4 +145,7 @@ def write_df_to_target(
) -> None:
"""
Saves dataframe to a specific target. |support_hooks|

.. versionchanged:: 0.9.0
Renamed ``write_df`` → ``write_df_to_target``
"""
42 changes: 40 additions & 2 deletions onetl/base/base_file_connection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,17 @@

class BaseFileConnection(BaseConnection):
"""
Implements generic methods for files and directories manipulation on some filesystem (usually remote)
Implements generic methods for files and directories manipulation on some filesystem (usually remote).

.. versionadded:: 0.8.0
"""

@abstractmethod
def path_exists(self, path: os.PathLike | str) -> bool:
"""
Check if specified path exists on remote filesystem. |support_hooks|
Check if specified path exists on remote filesystem. |support_hooks|.

.. versionadded:: 0.8.0

Parameters
----------
Expand All @@ -48,6 +52,8 @@ def is_file(self, path: os.PathLike | str) -> bool:
"""
Check if specified path is a file. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -76,6 +82,8 @@ def is_dir(self, path: os.PathLike | str) -> bool:
"""
Check if specified path is a directory. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -104,6 +112,8 @@ def get_stat(self, path: os.PathLike | str) -> PathStatProtocol:
"""
Returns stats for a specific path. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -132,6 +142,8 @@ def resolve_dir(self, path: os.PathLike | str) -> PathWithStatsProtocol:
"""
Returns directory at specific path, with stats. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -164,6 +176,8 @@ def resolve_file(self, path: os.PathLike | str) -> PathWithStatsProtocol:
"""
Returns file at specific path, with stats. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -196,6 +210,8 @@ def create_dir(self, path: os.PathLike | str) -> PathWithStatsProtocol:
"""
Creates directory tree on remote filesystem. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -229,6 +245,8 @@ def remove_file(self, path: os.PathLike | str) -> bool:

Supports only one file removal per call. Directory removal is **NOT** supported, use :obj:`~remove_dir` instead.

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -261,6 +279,8 @@ def remove_dir(self, path: os.PathLike | str, recursive: bool = False) -> bool:

If directory does not exist, no exception is raised.

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -305,6 +325,8 @@ def rename_file(

Supports only one file move per call. Directory move/rename is **NOT** supported.

.. versionadded:: 0.8.0

Parameters
----------
source_file_path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -353,6 +375,8 @@ def list_dir(
"""
Return list of child files/directories in a specific directory. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -400,6 +424,8 @@ def walk(

Just like :obj:`os.walk`, but with additional filter/limit logic.

.. versionadded:: 0.8.0

Parameters
----------
root : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -458,6 +484,8 @@ def download_file(

Supports only one file download per call. Directory download is **NOT** supported, use :ref:`file-downloader` instead.

.. versionadded:: 0.8.0

Parameters
----------
remote_file_path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -518,6 +546,8 @@ def upload_file(

Supports only one file upload per call. Directory upload is **NOT** supported, use :ref:`file-uploader` instead.

.. versionadded:: 0.8.0

Parameters
----------
local_file_path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -569,6 +599,8 @@ def read_text(self, path: os.PathLike | str, encoding: str = "utf-8") -> str:
r"""
Returns string content of a file at specific path. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -601,6 +633,8 @@ def read_bytes(self, path: os.PathLike | str) -> bytes:
"""
Returns binary content of a file at specific path. |support_hooks|

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -639,6 +673,8 @@ def write_text(

If file already exists, its content will be replaced.

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down Expand Up @@ -681,6 +717,8 @@ def write_bytes(self, path: os.PathLike | str, content: bytes) -> PathWithStatsP

If file already exists, its content will be replaced.

.. versionadded:: 0.8.0

Parameters
----------
path : str or :obj:`os.PathLike`
Expand Down
29 changes: 25 additions & 4 deletions onetl/base/base_file_df_connection.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,18 @@

class FileDFReadOptions(ABC):
"""
Protocol for objects supporting altering Spark DataFrameReader options
Protocol for objects supporting altering Spark DataFrameReader options.

.. versionadded:: 0.9.0
"""

@abstractmethod
def apply_to_reader(self, reader: DataFrameReader) -> DataFrameReader | ContextManager[DataFrameReader]:
"""
Apply provided format to :obj:`pyspark.sql.DataFrameReader`.

.. versionadded:: 0.9.0

Returns
-------
:obj:`pyspark.sql.DataFrameReader`
Expand All @@ -38,14 +42,18 @@ def apply_to_reader(self, reader: DataFrameReader) -> DataFrameReader | ContextM

class FileDFWriteOptions(ABC):
"""
Protocol for objects supporting altering Spark DataFrameWriter options
Protocol for objects supporting altering Spark DataFrameWriter options.

.. versionadded:: 0.9.0
"""

@abstractmethod
def apply_to_writer(self, writer: DataFrameWriter) -> DataFrameWriter | ContextManager[DataFrameWriter]:
"""
Apply provided format to :obj:`pyspark.sql.DataFrameWriter`.

.. versionadded:: 0.9.0

Returns
-------
:obj:`pyspark.sql.DataFrameWriter`
Expand All @@ -59,7 +67,9 @@ def apply_to_writer(self, writer: DataFrameWriter) -> DataFrameWriter | ContextM

class BaseFileDFConnection(BaseConnection):
"""
Implements generic methods for reading and writing dataframe as files
Implements generic methods for reading and writing dataframe as files.

.. versionadded:: 0.9.0
"""

@abstractmethod
Expand All @@ -70,6 +80,8 @@ def check_if_format_supported(
"""
Validate if specific file format is supported. |support_hooks|

.. versionadded:: 0.9.0

Raises
------
RuntimeError
Expand All @@ -80,12 +92,17 @@ def check_if_format_supported(
def path_from_string(self, path: os.PathLike | str) -> PurePathProtocol:
"""
Convert path from string to object. |support_hooks|

.. versionadded:: 0.9.0
"""

@property
@abstractmethod
def instance_url(self) -> str:
"""Instance URL"""
"""Instance URL.

.. versionadded:: 0.9.0
"""

@abstractmethod
def read_files_as_df(
Expand All @@ -98,6 +115,8 @@ def read_files_as_df(
) -> DataFrame:
"""
Read files in some paths list as dataframe. |support_hooks|

.. versionadded:: 0.9.0
"""

@abstractmethod
Expand All @@ -110,4 +129,6 @@ def write_df_as_files(
) -> None:
"""
Write dataframe as files in some path. |support_hooks|

.. versionadded:: 0.9.0
"""
4 changes: 4 additions & 0 deletions onetl/base/base_file_filter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,17 @@ class BaseFileFilter(ABC):
to determine if a file should be handled or not.

All filters are stateless.

.. versionadded:: 0.8.0
"""

@abstractmethod
def match(self, path: PathProtocol) -> bool:
"""
Returns ``True`` if path is matching the filter, ``False`` otherwise

.. versionadded:: 0.8.0

Examples
--------

Expand Down
12 changes: 12 additions & 0 deletions onetl/base/base_file_format.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,17 @@
class BaseReadableFileFormat(ABC):
"""
Representation of readable file format.

.. versionadded:: 0.9.0
"""

@abstractmethod
def check_if_supported(self, spark: SparkSession) -> None:
"""
Check if Spark session does support this file format. |support_hooks|

.. versionadded:: 0.9.0

Raises
------
RuntimeError
Expand All @@ -30,6 +34,8 @@ def apply_to_reader(self, reader: DataFrameReader) -> DataFrameReader | ContextM
"""
Apply provided format to :obj:`pyspark.sql.DataFrameReader`. |support_hooks|

.. versionadded:: 0.9.0

Returns
-------
:obj:`pyspark.sql.DataFrameReader`
Expand All @@ -44,13 +50,17 @@ def apply_to_reader(self, reader: DataFrameReader) -> DataFrameReader | ContextM
class BaseWritableFileFormat(ABC):
"""
Representation of writable file format.

.. versionadded:: 0.9.0
"""

@abstractmethod
def check_if_supported(self, spark: SparkSession) -> None:
"""
Check if Spark session does support this file format. |support_hooks|

.. versionadded:: 0.9.0

Raises
------
RuntimeError
Expand All @@ -62,6 +72,8 @@ def apply_to_writer(self, writer: DataFrameWriter) -> DataFrameWriter | ContextM
"""
Apply provided format to :obj:`pyspark.sql.DataFrameWriter`. |support_hooks|

.. versionadded:: 0.9.0

Returns
-------
:obj:`pyspark.sql.DataFrameWriter`
Expand Down
Loading