diff --git a/docs/python/classes.md b/docs/python/classes.md index b44fc73..d4df8ee 100755 --- a/docs/python/classes.md +++ b/docs/python/classes.md @@ -8,6 +8,20 @@ sidebar_label: Classes * Avoid inbuilt names. * Classes names should always be `PascalCase`. i.e. `MyClass` -* Even if you are building datatypes based on inbuilt class use PascalCase. i.e. `MyDict(dict):` -* Describe the class resposibility in name. +* **Abstract Classes and Mixins** + + Use [abstract containers](https://docs.python.org/3/library/collections.abc.html#module-collections.abc) if need to override datatypes. + - If you must build datatypes based on inbuilt class use PascalCase. i.e. `MyDict(dict):`. + + Use [`abc`](https://docs.python.org/3/library/abc.html) if you need pure OOP style abstract classes. Use `NotImplementedError` exceptions with overrides. + + mixin should be named with `Mixin` suffix such as `class LoginRequiredMixin` which can be used in multiple inheritance. +* Describe the class responsibility in name. * Custom Exceptions should always be named ending with `Error` i.e. `MyCustomError` + +##### Example + +```python +class HelloWorld: + pass + +class HelloWorldError(Exception): + pass +``` diff --git a/docs/python/docstrings.md b/docs/python/docstrings.md new file mode 100644 index 0000000..645e169 --- /dev/null +++ b/docs/python/docstrings.md @@ -0,0 +1,619 @@ +--- +id: docstrings +title: Convention for docstrings +sidebar_label: Docstrings +--- + +## Introduction + +> Code is more often read than written - Guido Van Rossum + +A few lines of description written on top of a function can save +the coder or a future developer hours or time reading the code. + +**Docstrings** are documentation that is written along with the code. +There are different typs of docstrings in Python: + +- Class +- Class methods +- Package +- Modules +- Functions +- Script + + +### Comments are not docstrings + +While comments are also written alongside code, docstrings are different +from comments. While comments start with a '#' symbol, docstrings are +also enclosed withing triple double quotes """This is a docstring""". +The placement of docstrings is also crucial. Docstrings placed arbitrarily +may simply be construed as a comment + +To illustrate this try the following in the python console + +```python +class Test: + """This is a class docstring + """ + + def example_method(): + """This is a method docstring + """ + pass + + def example_method_2(): + # This is a comment + pass +``` + + +```python +> print(Test.__doc___) +This is a class docstring +> +> print(Test.example_method.__doc__) +This is a method docstring +> +> print(Test.example_method_2.__doc__) +None +``` + +As you can see from the examples above, docstrings get +attached to the `__doc__` property of the code itself whereas, +the comments do not. + + +### Usage of docstrings + +From the console, you can use docstrings to an overview of +code as follows: + +```python +> help(Test) +Help on class Test in module __main__: + +class Test(builtins.object) + | This is a class docstring + | + | Methods defined here: + | + | example_method() + | This is a method docstring + | + | example_method_2() + + +``` + +If a docstring is provided, you can get more readable +information about python code. + +** They are also used by your IDE to give you information while developing.** + +Furthermore, there are tools that can take this to the +next level by creating a static website of documentation +for your code: + +- [Sphinx](http://www.sphinx-doc.org/en/stable/) +- [Epydoc](http://epydoc.sourceforge.net/) +- [Read the docs](https://readthedocs.org/) +- [Doxygen](http://www.stack.nl/~dimitri/doxygen/manual/docblocks.html#pythonblocks) +- [MkDocs](https://www.mkdocs.org/) +- [Pycco](https://pycco-docs.github.io/pycco/) + + +## Conventions + +The following sections describe the conventions that are recommended by Leapfrog + +### Python's official standard + +The PEPs are considered to be the official standard for Python. The following sections +talk specifically about docstring and we recommend that you read them + +- [PEP-256 -- Docstring Conventions](https://www.python.org/dev/peps/pep-0257/#specification) (**Must Read**) +- [PEP-8 Comments Section](https://www.python.org/dev/peps/pep-0008/#comments) +- [PEP 287 reStructedText Docstring Format](https://www.python.org/dev/peps/pep-0287/) + +### Docstring template selection + +While you can write anything between the triple quotes(""") to +write your docstring, it is generally recommended to follow a +template for consistency and also for libraries to be able to +parse your docstring easily. + +The official documentation standard for Python is ReStructed Text docstrings ([PEP 287](https://www.python.org/dev/peps/pep-0287/)). +However, Google docstrings have been widely accepted by the community as such we recommend it as we find it more readable and pythonic. +If you're using using Numpy related libraries, you should be using Numpy Docstrings + +- [Google docstrings](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) (**Recommended**) +- [reStructured Text Docstrings](http://docutils.sourceforge.net/rst.html) (Seen in many inbuilt python libraries.) +- [Numpy Docstrings](https://numpydoc.readthedocs.io/en/latest/format.html) (**Recommended for AI projects**) + + +### Where to add docstrings + +The developers should add docstrings in the following locations + +- At the start of every Python file +- At the beginning of every class +- After each function declaration +- At the beginning of the `__init__.py` file for Module/Package documentation +- In their tests to describe what they are testing. + +### What to not miss + +Use the documentation template of your choice and try not to miss the following +in the docstrings + +- A brief description of the entity that is being documented (**Mandatory**) +- Follow the above by above by examples and implementation details (**Recommended**) +- Add typing information where things can get confusing (**Recommended**). As python is a +dynamically typed language, adding some type definition to the documentation can save +the developers a lot of debugging time +- Autodeploy the documentation site using a static deployment tool +to check that your docstrings render correctly + +### Checking Docstring coverage + +- Use [pydocstyle](https://pydocstyle.pycqa.org/en/latest/) for linting your code against docstrings. When using `flake8`, [this](https://gitlab.com/pycqa/flake8-docstrings) plugin can be used. +- [interrogate](https://interrogate.readthedocs.io/en/latest/) (example below) is **recommended** to use for docstring coverage in the code. + + +``` + +================== Coverage for /Users/lynn/dev/interrogate/ ==================== +------------------------------------ Summary ------------------------------------ +| Name | Total | Miss | Cover | Cover% | +|---------------------------------------|---------|--------|---------|----------| +| src/interrogate/__init__.py | 1 | 0 | 1 | 100% | +| src/interrogate/__main__.py | 1 | 0 | 1 | 100% | +| src/interrogate/badge_gen.py | 5 | 0 | 5 | 100% | +| src/interrogate/cli.py | 2 | 0 | 2 | 100% | +| src/interrogate/config.py | 6 | 0 | 6 | 100% | +| src/interrogate/coverage.py | 25 | 0 | 25 | 100% | +| src/interrogate/utils.py | 10 | 0 | 10 | 100% | +| src/interrogate/visit.py | 15 | 0 | 15 | 100% | +| tests/functional/__init__.py | 1 | 0 | 1 | 100% | +| tests/functional/test_cli.py | 7 | 0 | 7 | 100% | +| tests/functional/test_coverage.py | 6 | 0 | 6 | 100% | +| tests/unit/__init__.py | 1 | 0 | 1 | 100% | +| tests/unit/test_badge_gen.py | 6 | 0 | 6 | 100% | +| tests/unit/test_config.py | 7 | 0 | 7 | 100% | +| tests/unit/test_utils.py | 13 | 0 | 13 | 100% | +|---------------------------------------|---------|--------|---------|----------| +| TOTAL | 106 | 0 | 106 | 100.0% | +---------------- RESULT: PASSED (minimum: 80.0%, actual: 100.0%) ---------------- + +``` + + + +### Examples + +#### Google Docstrings (recommended) + +```python + +"""Example Google style docstrings. + +This module demonstrates documentation as specified by the `Google Python +Style Guide`_. Docstrings may extend over multiple lines. Sections are created +with a section header and a colon followed by a block of indented text. + +Example: + Examples can be given using either the ``Example`` or ``Examples`` + sections. Sections support any reStructuredText formatting, including + literal blocks:: + + $ python example_google.py + +Section breaks are created by resuming unindented text. Section breaks +are also implicitly created anytime a new section starts. + +Attributes: + module_level_variable1 (int): Module level variables may be documented in + either the ``Attributes`` section of the module docstring, or in an + inline docstring immediately following the variable. + + Either form is acceptable, but the two should not be mixed. Choose + one convention to document module level variables and be consistent + with it. + +Todo: + * For module TODOs + * You have to also use ``sphinx.ext.todo`` extension + +.. _Google Python Style Guide: + https://google.github.io/styleguide/pyguide.html + +""" + +module_level_variable1 = 12345 + +module_level_variable2 = 98765 +"""int: Module level variable documented inline. + +The docstring may span multiple lines. The type may optionally be specified +on the first line, separated by a colon. +""" + + +def function_with_types_in_docstring(param1, param2): + """Example function with types documented in the docstring. + + `PEP 484`_ type annotations are supported. If attribute, parameter, and + return types are annotated according to `PEP 484`_, they do not need to be + included in the docstring: + + Args: + param1 (int): The first parameter. + param2 (str): The second parameter. + + Returns: + bool: The return value. True for success, False otherwise. + + .. _PEP 484: + https://www.python.org/dev/peps/pep-0484/ + + """ + + +def function_with_pep484_type_annotations(param1: int, param2: str) -> bool: + """Example function with PEP 484 type annotations. + + Args: + param1: The first parameter. + param2: The second parameter. + + Returns: + The return value. True for success, False otherwise. + + """ + + +def module_level_function(param1, param2=None, *args, **kwargs): + """This is an example of a module level function. + + Function parameters should be documented in the ``Args`` section. The name + of each parameter is required. The type and description of each parameter + is optional, but should be included if not obvious. + + If ``*args`` or ``**kwargs`` are accepted, + they should be listed as ``*args`` and ``**kwargs``. + + The format for a parameter is:: + + name (type): description + The description may span multiple lines. Following + lines should be indented. The "(type)" is optional. + + Multiple paragraphs are supported in parameter + descriptions. + + Args: + param1 (int): The first parameter. + param2 (:obj:`str`, optional): The second parameter. Defaults to None. + Second line of description should be indented. + *args: Variable length argument list. + **kwargs: Arbitrary keyword arguments. + + Returns: + bool: True if successful, False otherwise. + + The return type is optional and may be specified at the beginning of + the ``Returns`` section followed by a colon. + + The ``Returns`` section may span multiple lines and paragraphs. + Following lines should be indented to match the first line. + + The ``Returns`` section supports any reStructuredText formatting, + including literal blocks:: + + { + 'param1': param1, + 'param2': param2 + } + + Raises: + AttributeError: The ``Raises`` section is a list of all exceptions + that are relevant to the interface. + ValueError: If `param2` is equal to `param1`. + + """ + if param1 == param2: + raise ValueError('param1 may not be equal to param2') + return True + + +def example_generator(n): + """Generators have a ``Yields`` section instead of a ``Returns`` section. + + Args: + n (int): The upper limit of the range to generate, from 0 to `n` - 1. + + Yields: + int: The next number in the range of 0 to `n` - 1. + + Examples: + Examples should be written in doctest format, and should illustrate how + to use the function. + + >>> print([i for i in example_generator(4)]) + [0, 1, 2, 3] + + """ + for i in range(n): + yield i + + +class ExampleError(Exception): + """Exceptions are documented in the same way as classes. + + The __init__ method may be documented in either the class level + docstring, or as a docstring on the __init__ method itself. + + Either form is acceptable, but the two should not be mixed. Choose one + convention to document the __init__ method and be consistent with it. + + Note: + Do not include the `self` parameter in the ``Args`` section. + + Args: + msg (str): Human readable string describing the exception. + code (:obj:`int`, optional): Error code. + + Attributes: + msg (str): Human readable string describing the exception. + code (int): Exception error code. + + """ + + def __init__(self, msg, code): + self.msg = msg + self.code = code + + +class ExampleClass: + """The summary line for a class docstring should fit on one line. + + If the class has public attributes, they may be documented here + in an ``Attributes`` section and follow the same formatting as a + function's ``Args`` section. Alternatively, attributes may be documented + inline with the attribute's declaration (see __init__ method below). + + Properties created with the ``@property`` decorator should be documented + in the property's getter method. + + Attributes: + attr1 (str): Description of `attr1`. + attr2 (:obj:`int`, optional): Description of `attr2`. + + """ + + def __init__(self, param1, param2, param3): + """Example of docstring on the __init__ method. + + The __init__ method may be documented in either the class level + docstring, or as a docstring on the __init__ method itself. + + Either form is acceptable, but the two should not be mixed. Choose one + convention to document the __init__ method and be consistent with it. + + Note: + Do not include the `self` parameter in the ``Args`` section. + + Args: + param1 (str): Description of `param1`. + param2 (:obj:`int`, optional): Description of `param2`. Multiple + lines are supported. + param3 (list(str)): Description of `param3`. + + """ + self.attr1 = param1 + self.attr2 = param2 + self.attr3 = param3 #: Doc comment *inline* with attribute + + #: list(str): Doc comment *before* attribute, with type specified + self.attr4 = ['attr4'] + + self.attr5 = None + """str: Docstring *after* attribute, with type specified.""" + + @property + def readonly_property(self): + """str: Properties should be documented in their getter method.""" + return 'readonly_property' + + @property + def readwrite_property(self): + """list(str): Properties with both a getter and setter + should only be documented in their getter method. + + If the setter method contains notable behavior, it should be + mentioned here. + """ + return ['readwrite_property'] + + @readwrite_property.setter + def readwrite_property(self, value): + value + + def example_method(self, param1, param2): + """Class methods are similar to regular functions. + + Note: + Do not include the `self` parameter in the ``Args`` section. + + Args: + param1: The first parameter. + param2: The second parameter. + + Returns: + True if successful, False otherwise. + + """ + return True + + def __special__(self): + """By default special members with docstrings are not included. + + Special members are any methods or attributes that start with and + end with a double underscore. Any special member with a docstring + will be included in the output, if + ``napoleon_include_special_with_doc`` is set to True. + + This behavior can be enabled by changing the following setting in + Sphinx's conf.py:: + + napoleon_include_special_with_doc = True + + """ + pass + + def __special_without_docstring__(self): + pass + + def _private(self): + """By default private members are not included. + + Private members are any methods or attributes that start with an + underscore and are *not* special. By default they are not included + in the output. + + This behavior can be changed such that private members *are* included + by changing the following setting in Sphinx's conf.py:: + + napoleon_include_private_with_doc = True + + """ + pass + + def _private_without_docstring(self): + pass + +``` + +#### ReStructured Text Doc strings + +You can see this kind of docstring especially in python libraries. +Please use **Google** style as they are more readable. + +```python +"""The method below prints a given string twice + +The print method has been called twice for +implementing this method + +:param param1: String that is to be printed +:type param1: str +:return: Length of the input string +:rtype: int +""" +def print_twice(param1): + print(param1) + print(param1) + + return len(param1) + +"""The method below prints a given string twice. This is for type annotation. + +The print method has been called twice for +implementing this method + +:param str param1: String that is to be printed +:return: Length of the input string +:rtype: int +""" +def print_twice(param1: str) -> int: + print(param1) + print(param1) + + return len(param1) +``` + +## Doctests +:::caution +While this should not be used for all checks and [testing](testing.md) should be followed. They can be used for simple parameter checking. i.e. some default case like module level doctest in example below. + +They can be used with [pytest](https://docs.pytest.org/en/latest/doctest.html) as well. +::: + +Python provides tests through docstrings which can be leveraged by [doctests](https://docs.python.org/3/library/doctest.html). + +Example +```python +""" +This is the "example" module. + +The example module supplies one function, factorial(). For example, + +>>> factorial(5) +120 +""" + +def factorial(n): + """Return the factorial of n, an exact integer >= 0. + + >>> [factorial(n) for n in range(6)] + [1, 1, 2, 6, 24, 120] + >>> factorial(30) + 265252859812191058636308480000000 + >>> factorial(-1) + Traceback (most recent call last): + ... + ValueError: n must be >= 0 + + Factorials of floats are OK, but the float must be an exact integer: + >>> factorial(30.1) + Traceback (most recent call last): + ... + ValueError: n must be exact integer + >>> factorial(30.0) + 265252859812191058636308480000000 + + It must also not be ridiculously large: + >>> factorial(1e100) + Traceback (most recent call last): + ... + OverflowError: n too large + """ + + import math + if not n >= 0: + raise ValueError("n must be >= 0") + if math.floor(n) != n: + raise ValueError("n must be exact integer") + if n+1 == n: # catch a value like 1e300 + raise OverflowError("n too large") + result = 1 + factor = 2 + while factor <= n: + result *= factor + factor += 1 + return result + + +if __name__ == "__main__": + import doctest + doctest.testmod() +``` +This can be tested with: +``` +$python example.py -v +``` + +### References + +Thanks to the following + + +- [Pycharm Documentation on Docstrings](https://www.jetbrains.com/help/pycharm/using-docstrings-to-specify-types.html) +- [Documenting Python Code by RealPython](https://realpython.com/documenting-python-code/#documenting-your-python-projects) +- [PEP-256 -- Docstring Conventions](https://www.python.org/dev/peps/pep-0257/#specification) +- [PEP-8 Comments Section](https://www.python.org/dev/peps/pep-0008/#comments) +- [Documentation - Python Guide](https://docs.python-guide.org/writing/documentation/) +- [Documenting in Python - DevGuide](https://devguide.python.org/documenting/) +- [daouzli - stackoverflow](https://stackoverflow.com/questions/3898572/what-is-the-standard-python-docstring-format) +- [interrogate](https://interrogate.readthedocs.io/en/latest/) diff --git a/docs/python/environment_and_dependency.md b/docs/python/environment-and-dependency.md similarity index 67% rename from docs/python/environment_and_dependency.md rename to docs/python/environment-and-dependency.md index 5e1cac0..34311b5 100644 --- a/docs/python/environment_and_dependency.md +++ b/docs/python/environment-and-dependency.md @@ -1,12 +1,12 @@ --- -id: environment_and_dependency +id: environment-and-dependency title: Environment Isolation and Dependency Management sidebar_label: Environment Isolation and Dependency Management --- #### Information on development environment and dependency: -##### Environment Isolation: +### Environment Isolation: * System installed `python` should never be used for development. Isolate your development. * Any of the following can be used for python isolation: @@ -17,8 +17,8 @@ sidebar_label: Environment Isolation and Dependency Management -##### Dependency Management: +### Dependency Management: -* [poetry](https://python-poetry.org/) is recommended. - - If not `poetry` use `pip` with `requirements.txt` -* You can use `setuptools` and `setup.py` as well for requirements handling. They **must** be used for installable modules. +* [poetry](https://python-poetry.org/) is recommended as it handles dependency as well as build system. +* You can use `setuptools` and `setup.py` as well for requirements handling through `requires`. They **must** be used for install-able modules. +* `requirements.txt` style should be avoided although you may come across this style a lot. diff --git a/docs/python/exceptions.md b/docs/python/exceptions.md new file mode 100644 index 0000000..a2b7f21 --- /dev/null +++ b/docs/python/exceptions.md @@ -0,0 +1,41 @@ +--- +id: exceptions +title: Exception Handling +sidebar_label: Exception Handling +--- + +#### The following convention should be followed for `Exception` handling: + +* `Exception` handling is a must and should be mitigated. +* Do not use bare `except` or `except Exception` which catches all the exception. + - Always be specific on exception. E.g. catch only `FileNotFoundError` if you are say moving a file. +* **User Defined Exceptions**: + - Write your custom error only when your error is not described or fulfilled by [internal exceptions](https://docs.python.org/3/library/exceptions.html). + - Create custom `Exception` class primarily suffixing it with `Error` such as `MyCustomError(Exception)` and use it. + - Always use `Exception` as your parent class for user defined exceptions. Donot use `BaseException`. +* Add traceback to your mitigation. i.e. either `logging` or mails. Donot `pass`. +* The `try` block should be specific to desired exception. Donot use huge code chunk in `try`. Use `else` if needed. +```python + try: + value = int(some_str) + except ValueError: # when not using exception object + WHEN some_str IS not valid as value. + except TypeError: + WHEN some_str IS OTHER TYPE such as [], () + else: #can be avoided if exceptions are handled and returned. This is in context to try block. + DO SOMETHING WITH VALUE + + try: + value = int(some_str) + except (ValueError, TypeError) as error: + HANDLE BOTH exception and use exception object + else: # If needed + do something when try succeeds. + finally: + codebase to run anyway +``` +* `finally` can be used if you need to run the block whatever the case. `context` can be used in many cases to avoid `finally`. +* `sys.exc_info` and [`traceback`](https://docs.python.org/3/library/traceback.html) can be used for traceback. +* Please read [this](https://cosmicpercolator.com/2016/01/13/exception-leaks-in-python-2-and-3/) on exceptions handling internals and leaks when referencing exceptions. + + diff --git a/docs/python/files.md b/docs/python/files.md index bb0cef6..032d7b2 100755 --- a/docs/python/files.md +++ b/docs/python/files.md @@ -7,7 +7,7 @@ sidebar_label: Files, Folders & Modules #### The following convention should be followed for files, folders and package naming: * Name in `snake_case` or descriptive single words all in **lowercase**. E.g. `helper.py` or `sftp_fetcher.py` or `tools` -* Be explicit and descriptive of their functionality. Donot have short and ambigous file and folder names. +* Be explicit and descriptive of their functionality. Donot have short and ambiguous file and folder names. - E.g. `utils.py` or `utils` will describe of utility. - E.g. `aws_helper.py` will describe helper related to AWS. * Donot Clash names with inbuilt and famous modules. diff --git a/docs/python/functions.md b/docs/python/functions.md index 397897d..22cd959 100755 --- a/docs/python/functions.md +++ b/docs/python/functions.md @@ -12,3 +12,21 @@ sidebar_label: Functions * for bound methods in class `self` should be used for first argument. * for class methods in class `cls` should be used for first argument. * `decorators` should be named in function convention. + + +```python +def get_db_connection(username, db_name): + return connection + +#method + +def get_db_connection(self, username, db_name): + return connection + + +# classmethod +@classmethod +def multiple_param_initializer(cls, cls_param): + return cls(cls_param) + +``` diff --git a/docs/python/general.md b/docs/python/general.md index 645c0da..17d588d 100644 --- a/docs/python/general.md +++ b/docs/python/general.md @@ -6,40 +6,69 @@ sidebar_label: General Coding Guidelines #### These are the general guidelines to be followed: -* Always use `python3` and try to stay above version `3.5`. **Latest stable** is recommended. +* See [tools](tools.md) that can be used in development environment setup to ease your coding process. +* Always use `python3` and try to stay above version `3.5`. **Latest stable** is always recommended. * Indentation should always be **space** and width should always be **4**. -* `Docker` can be used for deployment. Use `python` images for [docker](https://hub.docker.com/_/python). +* File size and functionality: + - break files into modules if you feel they have multiple functionalities. + - Always try to adhere to single functionality or single class per file. + - Set a goal of approximate linenumbers per file. +* Always go for pythonic syntax + - comprehensions over `map` and loop + - minimal use of `lambda`, use `operator` module with keyfunctions in `sorted`, `groupby` etc. + - ternary with `if else` in same line. Donot use `and or` clause. i.e. `value and req_value or default`. +* Imports: + - Always `import` specific namespace. + - Try to use parent namespace for actions. i.e. `import sys: sys.path` or `import sys.path` rather that `from sys import path` + - Never use `import *` i.e. `from MODULE import *` + - use namespace whenever possible. Use `as SOMEOTHERNAMESPACE` for collision of namespace +* If you are using `else` with loops that has `break`. Just comment `#nobreak` for reference as it is for that usage. See [this](http://python-notes.curiousefficiency.org/en/latest/python_concepts/break_else.html) for some clarity. + ```python + for each in each_collection: + if condition: + break + else: #nobreak + WHEN break DOESNOT HAPPEN + + while True: + if condition: + break + else: #nobreak + WHEN break DOESNOT HAPPEN + ``` +* Use `pathlib` for path related use case rather than `os.path` +* Use `mypy` and type annotation when possible for type safe code. +* `Docker` can be used for deployment. Use `python` images for [`docker`](https://hub.docker.com/_/python). * Use `generators` and `yield` instead of data structures for high streams of data. -* Use `itertools`, `functools` and `collection` modules to ensure we are using right data structures and utilities. -* Use `dataclasses` if available. Go for `attrs` library if `dataclass` is not present. -* Use `is not` and `is` for `None`, `True` and `False` check only. +* Use `itertools`, `functools` for utilities and `collections` for data structures. +* Use `is not` and `is` for `None`, `True` and `False` specific check only. If most cases truthy and falsy can be checked with `if VARNAME:`. * Strings: - - Adhere to one quote practice. Double quote is recommended. Python doesnot differentiate between *"'"* or *'"'* + - Adhere to one quote practice. Double quote is recommended. Python doesnot differentiate between **'** or **"**. This may be controlled by `formatter` used as well. - Should be interpolated with either [fstring](https://www.python.org/dev/peps/pep-0498/) or `.format` methods. Try to avoid `%`. - - Use `join` method for concatenation instead of `+` -* `logging` is always a must. Use the following levels as required: - - **DEBUG**: log parameters and arguments. Information needed when we need to debug or develop. Should be avoided in production. - - **INFO**: log basic information such as function entry, file being processed et al - - **WARN**: log user security and other warnings that are not critical - - **ERROR**: error related logs. Use exception method to log tracebacks in case of exceptions. - - **CRITICAL**: blocking issues or immediate attention issues. - - **ERROR and CRITICAL** levels should be mitigated and informed. - - `logger` is used for naming logger. - - It is singleton and single threaded by default for `a name` of the logger. Can be non-blocking if required. -* `Exception` handling is a must along with logging. - - Do not use bare `except` or `except Exception` which catches all the exception. Be specific on exception. E.g. catch only `FileNotFoundError` if you are say moving a file. - - For modules specific error, if something internal is not fulfilling then try to create custom `Exception` class primarily naming it Error such as `MyCustomError(Exception)` and use it. -* Use `context` whenever supported especially for io related actions. + + `format_map` should be used for key mapped formatting. + - `+` can be used for direct string concatenation. Use `join` method for concatenation instead of `+=` when iterating. +* Use `context` whenever supported especially for io related closing actions. - i.e. `with` statement when supported. - Always remember to close on exit. i.e. if you open the file `close` on `finally` or better use `with` or `contextlib.closing`. +* **OOP** + - While `python` is an OOP, you can always choose `functions` and `modules` over `class` if there is only one `object` to be created like `Singletons`. + - Use [`abc`](https://docs.python.org/3/library/abc.html) if you need abstraction. Mixins are more famous in python due to multiple inheritance. + - Use `property` setter getter only when you need readonly attributes. `__` variables can be used for some privacy. + - Use `super` for overrides and parent calls. + - Use inbuilt `dataclasses` if available. Go for `attrs` library if `dataclasses` is not present or you require a much richer library. + - Use `classmethod` decorator for multiple initialization of classes as well as `staticmethod` where needed. * Use `pdb` as debugger whenever required. * Multi-threading can be especially used when we have io bound and network bound multiple operation. Multiprocessing can be used to use multiple cores. - Recommended module is `concurrent.futures` in most cases. If lower level API is needed there is always `threading` and `multiprocessing` module. - - Use `asyncio` for IO bound async codes. This is something new and constantly changing in `python`. - - Be very carefult on threads and locks, so always discuss what you are doing. + - Be very carefult on threads and locks, so always discuss what you are doing as it may not always be optimized. + - Use `asyncio` for IO bound async flow. This is something new and constantly changing in `python`. +* Try to use configurations outside python files. Usually they are not git tracked so should be editable by others. Try `settings.py` or `config.py` if you **must**. This cannot be the case for frameworks. * Recommended third party modules: - - `sqlalchemy` for ORM related database stuffs. + - For Relational Database: + + Use `sqlalchemy` [core](https://docs.sqlalchemy.org/en/13/core/) for DB abstraction. This is particularly helpful when doing testing in `sqlite` and some other database for production. Also, for query and parameters consistency. + + Use `sqlalchemy` [ORM](https://docs.sqlalchemy.org/en/13/orm/) or framework supported **ORM** when using specific framework. + + Use DBAPI drivers such as `pyodbc`, `sqlite`, `mysqlclient` etc only when you donot want `sqlalchemy` dependency or when you are very performance conscious. While the API will be mostly compatible for this as python has DBAPI specification. Parameters binding and some methods may be incompatible or unavailable. **sqlalchemy core is recommended.** - `requests` for http request stuff. - - `attrs` for data oriented objects and class. - - `pytest` for tests. - + + `aiohttp` or `httpx` are also good. + - `attrs` for data oriented objects and classes design. If you don't want to use `dataclasses`. + - `pytest` for tests. diff --git a/docs/python/logging.md b/docs/python/logging.md new file mode 100644 index 0000000..0a15057 --- /dev/null +++ b/docs/python/logging.md @@ -0,0 +1,20 @@ +--- +id: logging +title: Logging +sidebar_label: Logging Convention +--- + +#### The following convention should be followed for using `logging` in python: + +* Inbuilt `logging` module is used in most cases. You can look into [structlog](https://www.structlog.org/en/stable/) for more granular logging. +* `logging` is always a must. Use the following levels as required: + - **DEBUG**: log parameters and arguments. Information needed when we need to debug or develop. Should be avoided in production. + - **INFO**: log basic information such as function entry, file being processed et al + - **WARN**: log user security and other warnings that may require attention or may need to be avoided. + - **ERROR**: errors in programs. + - **CRITICAL**: blocking issues or immediate attention issues. +* `logger` is used for naming single logger object. Use `NAME_logger` name for more than one logger when required. +* It is singleton and single threaded by default for given name of the logger. Can be [non-blocking](https://docs.python.org/3/howto/logging-cookbook.html#dealing-with-handlers-that-block) if required. +* See [Logging Cookbook](https://docs.python.org/3/howto/logging-cookbook.html) for reference. +* **ERROR and CRITICAL** levels should be mitigated and informed. + - Always use `exception` method rather than `error` method of `logger` object to log traceback when catching exceptions. diff --git a/docs/python/project-structure.md b/docs/python/project-structure.md new file mode 100644 index 0000000..eb6258c --- /dev/null +++ b/docs/python/project-structure.md @@ -0,0 +1,57 @@ +--- +id: project-structure +title: Project Structure and Templates +sidebar_label: Project Structure +--- + +#### The following folder structure should be used for projects: + + +* :file_folder: Project Root: + - :memo: pyproject.toml + - :memo: setup.py `and` :memo: setup.cfg (**OPTIONAL**) + +:::info +* If not using `poetry`, `pyproject.toml` file can be omitted as `setup.py` can have dependencies as well. +* `pyproject.toml` be used for [alternate build system](https://www.python.org/dev/peps/pep-0518/) if needed. +::: + +* + - :file_folder: docs + * Your documentation + - :file_folder: bin (**OPTIONAL**) => For entrypoint scripts which calls {PROJECT_NAME} + + This can be generated using `setup.py` as well. + - :file_folder: data (**OPTIONAL**) + * Data for project. + +:::tip +* This folder structure can be used as well. It will have **src** folder for more descriptive stature. + + :file_folder: src + - :file_folder: {PROJECT_NAME} + +::: +* + - :file_folder: {PROJECT_NAME} + + :memo: `__init__.py` + + :memo: `__main__.py` (**OPTIONAL**) => for calling with `python -m` + + :file_folder: utils + + :file_folder: service + + :file_folder: config + - :file_folder: tests + + :file_folder: test of projects + + :memo: conftest.py + - :memo: LICENSE (**OPTIONAL**) + - :memo: README (can be `md` or `rst`) + - :file_folder: :memo: Other files/folders from third parties (**OPTIONAL**) such as tox.ini + +:::note ++ **There can be cases where MVC folder structure as well as framework related folder structure can be used.** + - The framework recommended structure should be followed in such case. ++ The OOP style cases of class as filename structue is not always necessary or recommended but can be used if needed. +::: + + +### Project Template +* [Python Package Authority Sample](https://github.com/pypa/sampleproject) can be used as bootstrap. +* Look into [cookiecutter](https://cookiecutter.readthedocs.io/en/1.7.2/) tool for template generation. + - [List of templates for cookiecutter.](http://cookiecutter-templates.sebastianruml.name/) diff --git a/docs/python/project_structure.md b/docs/python/project_structure.md deleted file mode 100644 index 303fc73..0000000 --- a/docs/python/project_structure.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -id: project_structure -title: Project Structure and Templates -sidebar_label: Project Structure ---- - -#### The following folder structure should be used for projects: - - -* :file_folder: Project Root: - - :memo: pyproject.toml (can be requirements.txt and setup.py) - - :file_folder: docs - * Your documentation - - :file_folder: data - * Data for project. - - :file_folder: {PROJECT_NAME} - + :file_folder: utils - + :file_folder: service - + :file_folder: config - - :file_folder: tests - + test of projects - - :memo: LICENCE (**OPTIONAL**) - - :memo: README (can be `md` or `rst`) - - Other configurable from third parties (**OPTIONAL**) such as tox.ini - - - -Please look into [cookiecutter](https://cookiecutter.readthedocs.io/en/1.7.2/) for template generation which gives a lot of options. diff --git a/docs/python/testing.md b/docs/python/testing.md new file mode 100644 index 0000000..bfd4390 --- /dev/null +++ b/docs/python/testing.md @@ -0,0 +1,25 @@ +--- +id: testing +title: Testing in python +sidebar_label: Testing +--- + +#### Test is integral part of sofware quality and should not be missed. + + + +* Always use `pytest` for testing codes. + + `unittest` provided by standard python can be used too if there is some blockage in using `pytest`. +* `tox` and `nox` are vey good tools especially for CI and multiple version tests. +* `hypothesis` and `mock` can be used for faking data and property testing. +* Testing should be broken to `unit` as well as `functional`. +* Use `coverage` to alert yourself of test coverage. Keep a target of **80 % - 90 %** coverage if **100%** is not achieved. +* Only test the changes you made or functionality you added when testing a codebase of well known frameworks. +* `selenium` as well as `webtest` can be used for web based API testing. +* `jsonschema` and `genson` like tool can be used for JSON validity. +* Always confirm the `schema` when testing Web API response data. +* Passing tests for `merge` should be priority for all projects. +* Tests should always cover: + + **Unit**: for your code units. Please use `mock` for external dependency and side effects. + + **Functional**: Your program functionality. + + **Integration**: Your whole program integration. diff --git a/docs/python/tools.md b/docs/python/tools.md index f273be1..a54f592 100644 --- a/docs/python/tools.md +++ b/docs/python/tools.md @@ -1,31 +1,45 @@ --- id: tools -title: Tools to use for easing development and maintaining consistency. +title: Tools to use for easing development and maintaining consistency. There are links to readings as well. sidebar_label: Tools and Libraries ---- +--- -### Multiple tools on indented list are options with first one being recommended. +:::info +These tools can be used along with your development environment and IDE so that you can follow coding conventions better. +::: -#### Templates: +### Templates: * [cookiecutter](https://cookiecutter.readthedocs.io/en/1.7.2/) -#### Dependency Management: +### Dependency Management: * [poetry](https://python-poetry.org/) -#### Linters: +### Linters: * [flake8](https://flake8.pycqa.org/en/latest/) with [plugins](https://github.com/DmytroLitvinov/awesome-flake8-extensions) - * [pylint](https://www.pylint.org) + * Alternative: [pylint](https://www.pylint.org) +* [bandit](https://bandit.readthedocs.io/en/latest/) to find common security issues. This can be used with `flake8` as a [plugin](https://pypi.org/project/flake8-bandit/) -#### Formatters +### Formatters: * [black](https://black.readthedocs.io/en/stable/) - * [autopep8](https://pypi.org/project/autopep8/) - * [yapf](https://pypi.org/project/yapf/) - -#### Testing + - Alternative: [autopep8](https://pypi.org/project/autopep8/) + - Alternative: [yapf](https://pypi.org/project/yapf/) +* [isort](https://timothycrosley.github.io/isort/) for sorting only imports in codes. This is OPTIONAL. + - Alternative: [reorder-python-imports](https://github.com/asottile/reorder_python_imports) another way of sorting imports. + +### Testing: * [pytest](https://pytest.org) with [plugins](https://docs.pytest.org/en/2.7.3/plugins_index/index.html) - * Inbuilt `unittest` + - Alternative: Inbuilt `unittest` +* [hypothesis](https://hypothesis.readthedocs.io/en/latest/) and `mock` for data generation and mocking in tests. +* [tox](https://tox.readthedocs.io/en/latest/) for test automation in different `python` version + - Alternative: [nox](https://nox.thea.codes/en/stable/) is `tox` with `python` API i.e. py file settings so can be used if you need any dynamism. -#### Other tools -* [coverage](https://coverage.readthedocs.io/en/coverage-5.1/) for testing code coverage. +### Other tools: +* [coverage](https://coverage.readthedocs.io/en/coverage-5.1/) for checking code coverage of tests. * [interrogate](https://interrogate.readthedocs.io/en/latest/) for docstring coverage check. -* [hypothesis](https://hypothesis.readthedocs.io/en/latest/) and `mock` for data. +* [mypy](http://mypy-lang.org/index.html) optional static type coding with python through annotations. + +### Readings and References: +* [Design Patterns](https://python-patterns.guide/) +* [Official Documentation](https://docs.python.org/3/) +* [Python Code Quality](https://meta.pycqa.org/en/latest/index.html) for the tools like `flake8`, `pylint`, `bandit` etc along with others. +* [Python Packages](https://www.pypa.io/en/latest/) diff --git a/docs/python/variables.md b/docs/python/variables.md index 711f64a..c1607e7 100755 --- a/docs/python/variables.md +++ b/docs/python/variables.md @@ -8,12 +8,12 @@ sidebar_label: Variables * `snake_case` or descriptive word all in **lowercase** for any type of variables except `CONSTANTS`. * `ALL_CAPS` for constants. `python` doesnot have the concept of constants so this is just a convention. -* Variable Privacy - - `__{variable_name}` if you want something to be private. - - `_{variable_name}` if you want something to be not publicly used or something that may change later. +* Variable Privacy is not something that python encourages or supports but there may be case where privacy is required. + - `__{variable_name}` if you want something to denote private. + - `_{variable_name}` if you want something to denote not publicly used or something that may change later. - `__{variable_name}` are not directly accesible while `_{variable_name}` are. They are just for convention. * Avoid builtin variable clash. Especially in `globals`. You can attach `_` as suffix to builtin names if you deem the name necessary for your variable. - `all` or `id` is very tempting variable names but they are builtin methods in python. Go with `all_` or `id_` for these or better yet choose something else as a name. * While it is tempting to use `i`, `k`, `v` and `f` especially in contexts and for loops. Please avoid them. - use `key`, `value`, `index` instead. - - use descriptive contexts such as `with open(FILENAME) as open_file: pass` rather that `with open(FILENAME) as f:` + - use descriptive contexts such as `with open(FILENAME) as open_file: pass` rather than `with open(FILENAME) as f: pass` diff --git a/sidebars.js b/sidebars.js index 161b186..2a17f10 100644 --- a/sidebars.js +++ b/sidebars.js @@ -39,6 +39,17 @@ module.exports = "git/code_review_checklist" ], "Python": [ + "python/environment-and-dependency", + "python/project-structure", + { + "type": "category", + "label": "Practices and Tools", + "items": [ + "python/general", + "python/tools" + ] + }, + "python/docstrings", { "type": "category", "label": "Naming Convention", @@ -49,10 +60,9 @@ module.exports = "python/classes" ] }, - "python/tools", - "python/general", - "python/environment_and_dependency", - "python/project_structure" + "python/exceptions", + "python/logging", + "python/testing" ], "JavaScript": [ {