docs: 📝

CONTRIBUTING.md

@@ -10,7 +10,7 @@ In addition to the standards outlined in PEP 8, we have a few guidelines:
 
 * Max line length
 
-  Line-length can exceed 79 characters, we set max-line-length to 160 to ignore most of line-length warning,
+  Line-length can exceed 79 characters, we set max-line-length to 120 to ignore most of line-length warning,
   But try to keep the line-length within 100 characters.
 
 * Import order and grouping
@@ -26,23 +26,7 @@ In addition to the standards outlined in PEP 8, we have a few guidelines:
 
   See [this code](biothings/utils/es.py) as an example.
 
-* Inline comments
-
-  When you have multiple inline comments, try to align them by groups:
-
-  **Yes**
-
-        self.number_of_shards = number_of_shards            # set number_of_shards when create_index
-        self.number_of_replicas = int(number_of_replicas)   # set number_of_replicas when create_index
-        self.step = step   # the bulk size when doing bulk operation.
-        self.s = None      # number of records to skip, useful to continue indexing after an error.
-
-  **No**
-
-        self.number_of_shards = number_of_shards    # set number_of_shards when create_index
-        self.number_of_replicas = int(number_of_replicas)    # set number_of_replicas when create_index
-        self.step = step    # the bulk size when doing bulk operation.
-        self.s = None    # number of records to skip, useful to continue indexing after an error.
+  Note: You can use [isort](https://pypi.org/project/isort/) package or [its vscode extension](https://github.com/microsoft/vscode-isort) to format import order and grouping easily.
 
 * docstring format
 
@@ -84,32 +68,75 @@ particular guideline from [PEP8 doc](https://pep8.org/#a-foolish-consistency-is-
 
 ### Check your code style
 
-We recommand to setup [flake8](http://flake8.pycqa.org) as your code-style checker with your editor.
-Fix the styling issue before you push the code to the github.
+We recommand to setup [fourmat](https://github.com/4Catalyzer/fourmat) as your code-style checker/formmater with your editor.
+Fix the styling issue before you push the code to the github. [fourmat](https://github.com/4Catalyzer/fourmat) conviniently combines
+three code-styling tools: [flake8](http://flake8.pycqa.org), [black](https://github.com/python/black) and [isort](https://pypi.org/project/isort/).
+
+We have already included config files for code styling check and formatting: [.flake8](.flake8) for flake8 and [pyproject.toml] for black and isort,
+so that we all use the same settings when run `fourmat`.
+
+You can check out more config options for each tool:
+
+* [flake8 configuration](http://flake8.pycqa.org/en/latest/user/configuration.html)
+* [black configuration](https://black.readthedocs.io/en/stable/usage_and_configuration/the_basics.html#configuration-via-a-file)
+* [isort configuration](https://github.com/microsoft/vscode-isort#settings)
+
+#### run fourmat in CLI
+
+* install fourmat
+
+  ```bash
+  pip install fourmat
+  ```
+
+* check code
+
+  ```bash
+  fourmat check <file_or_folder>
+  ```
 
-You can use this as your [flake8](http://flake8.pycqa.org) [config file](http://flake8.pycqa.org/en/latest/user/configuration.html):
+* format code
 
-    [flake8]
-    # ignore=E226,E265,E302,E402,E731,F821,W503
-    max-line-length=160
+  ```bash
+  fourmat fix <file_or_folder>
+  ```
 
-There are project-level [flake8](http://flake8.pycqa.org) settings provided in this top-level [setup.cfg](setup.cfg) file, so you can just run `flake8 <your file>` to check your code, or explicitly `flake8 --config ./setup.cfg <your file>`.
+#### setup black and isort formatter in VSCode
 
-In VSCode, you can add this to settings.json:
+* install both [Black Formatter](https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter) and
+  [isort](https://marketplace.visualstudio.com/items?itemName=ms-python.isort) vscode extensions
 
-    "python.linting.flake8Args": [
-        "--max-line-length=160",
-        //"--ignore=E226,E265,E302,E402,E731,F821,W503"
-            //E226  Missing whitespace around arithmetic operator
-            //E265  block comment should start with '# '
-            //E302  Expected 2 blank lines, found 0
-            //E402  Module level import not at top of file
-            //E731  Do not assign a lambda expression, use a def
-            //F821  Undefined name name
-            //W503  Line break occurred before a binary operator
-    ],
+* setup auto formatting on file dave:
 
-The above configuration includes an example of ignoring certain errors or warnings. You may include them if needed, but we recommend to [ignore specific errors/warnings at the specific code](https://flake8.pycqa.org/en/3.1.1/user/ignoring-errors.html#in-line-ignoring-errors) instead.
+  Add this settings to your vscode's `settings.json` file (up to you to set it at the project, user or workspace level):
+
+  ```json
+  "[python]": {
+      "editor.defaultFormatter": "ms-python.black-formatter",
+      "editor.formatOnSave": true,
+      "editor.codeActionsOnSave": {
+          "source.organizeImports": true
+      },
+  },
+  "isort.args":["--profile", "black"],
+  ```
+
+#### Ignore some code-styling check errors
+
+For some particular cases, if you think the reported errors are false alerts, you can [ignore specific errors/warnings at the specific code](https://flake8.pycqa.org/en/latest/user/violations.html#ignoring-violations-with-flake8) instead. A few common examples below:
+
+```python
+# ignore one or muliple errors
+example = lambda: 'example'  # noqa: E731
+
+example = lambda: 'example'  # noqa: E731,E123
+
+# ignore all errors, but not recommended, better to include specific errors
+example = lambda: 'example'  # noqa
+
+# ignore the whole file if this is at the top:
+# flake8: noqa
+```
 
 ### Other recommended but not required style checks
 
@@ -119,31 +146,25 @@ You can always do extra checks on your code before commits. Some checkers may gi
 
   You may find these flake plugins can be very useful (install them using `pip`):
 
-      pip install flake8-builtins flake8-comprehensions flake8-logging-format pep8-naming flake8-import-order
+      pip install flake8-builtins flake8-comprehensions flake8-logging-format pep8-naming
 
 * [PyLint](https://www.pylint.org/)
 
       pip install pylint
 
   When there are conflicts with our style guidelines above from these extra checkers, follow our own rules.
 
-### Python code formatter
-
-Some Python code formatters can be useful, but use them with caution. Double check the converted code to make sure you don't break your own code.
-
-* [fourmat](https://github.com/4Catalyzer/fourmat):  Flake8 + Black + isort = ❤️
-* [black](https://github.com/python/black)
-* [yapf](https://github.com/google/yapf/)
-
 ### Setup pre-commit
 
-After all above recommendations, we already setup [pre-commit](https://github.com/pre-commit/pre-commit) framework.
-You just need to install it on your local git repo:
+Optionally, you can also setup [pre-commit](https://github.com/pre-commit/pre-commit) for your
+local repository. We have included a [.pre-commit-config.yaml](.pre-commit-config.yaml) file in this repository.
+You just need to install it on your local git repository:
 
-      pre-commit install
+    pre-commit install
 
 ### Some useful references
 
+* [fourmat](https://github.com/4Catalyzer/fourmat):  Flake8 + Black + isort = ❤️
 * [The list of Flake8 rules](https://lintlyci.github.io/Flake8Rules/)
 * [The Black code style](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html)
 * [**requests** code style](http://python-requests.org//en/latest/dev/contributing/#kenneth-reitz-s-code-style)

README.md

@@ -1,3 +1,5 @@
+# BioThings SDK
+
 [![Downloads](https://pepy.tech/badge/biothings)](https://pepy.tech/project/biothings)
 [![biothings package](https://badge.fury.io/py/biothings.svg)](https://pypi.python.org/pypi/biothings)
 [![biothings_version](https://img.shields.io/pypi/pyversions/biothings.svg)](https://pypi.python.org/pypi/biothings)
@@ -8,8 +10,6 @@
 [![Tests Status](https://github.com/biothings/biothings.api/actions/workflows/run-tests.yml/badge.svg)](https://github.com/biothings/biothings.api/actions/workflows/run-tests.yml)
 [![Documentation Status](https://readthedocs.org/projects/biothingsapi/badge/?version=latest)](https://docs.biothings.io/en/latest/?badge=latest)
 
-# BioThings SDK
-
 ## Quick Summary
 
 BioThings SDK provides a Python-based toolkit to build high-performance data APIs (or web services) from a single data source or multiple data sources. It has the particular focus on building data APIs for biomedical-related entities, a.k.a "BioThings" (such as genes, genetic variants, drugs, chemicals, diseases, etc.).
@@ -67,17 +67,48 @@ update API frontends.
 You can install the latest stable BioThings SDK release with pip from
 [PyPI](https://pypi.python.org/pypi), like:
 
+    # default to install web requirements only for running an API
     pip install biothings
+    # or include additional requirements useful for running an API on production
+    # like msgpack, sentry-sdk pacakages
+    pip install biothings[web_extra]
+
+    # install hub requirements for running a hub (including CLI)
+    pip install biothings[hub]
+
+    # install CLI-only requirements if you only need CLI for develop a data plugin
+    pip install biothings[cli]
+
+    # need support for docker data plugin
+    pip install biothings[hub,docker]
+    # or if use ssh protocol to connect to a remote docker server
+    pip install biothings[hub,docker_ssh]
+
+    # just install everything for dev purpose
+    pip install biothings[dev]
+
+You can check more details for the optional dependecy packages directly in [setup.py](setup.py) file.
 
 You can install the latest development version of BioThings SDK directly
 from our github repository like:
 
     pip install git+https://github.com/biothings/biothings.api.git#egg=biothings
 
+    # from a branch or commit
+    pip install git+https://github.com/biothings/biothings.api.git@0.12.x#egg=biothings
+
+    # include optional dependecies
+    pip install git+https://github.com/biothings/biothings.api.git@0.12.x#egg=biothings[web_extra]
+
+    # can be very useful to install in “editable” mode:
+    pip install -e git+https://github.com/biothings/biothings.api.git@0.12.x#egg=biothings[web_extra]
+
 Alternatively, you can download the source code, or clone the [BioThings
 SDK repository](https://github.com/biothings/biothings.api) and run:
 
-    python setup.py install
+    pip install .
+    # or
+    pip install .[web_extra]
 
 ## Get started to build a BioThings API
 
@@ -88,4 +119,5 @@ We recommend to follow [this tutorial](https://docs.biothings.io/en/latest/tutor
 The latest documentation is available at https://docs.biothings.io.
 
 ## How to contribute
-Please check out this [Contribution Guidelines](CONTRIBUTING.md) and [Code of Conduct](CODE_OF_CONDUCT.md) document.
+
+Please check out this [Contribution Guidelines](CONTRIBUTING.md) and [Code of Conduct](CODE_OF_CONDUCT.md) document.

RELEASE.md

@@ -1,53 +1,55 @@
-### This is the procedure we use for "biothings" package release
+# Release instructions
+
+## This is the procedure we use for "biothings" package release
 
 1. requires both `wheel` and `twine` packages installed
 
-    ```bash
-    pip install wheel twine
-    ```
+   ```bash
+   pip install wheel twine
+   ```
 
 2. Update version number in [biothings/__init__.py](biothings/__init__.py).
 
 3. Check and update [setup.py](setup.py) if needed (dependencies, metadata etc.).
 
 4. Build the package locally:
 
-    ```bash
-    python setup.py sdist bdist_wheel
-    ```
+   ```bash
+   python setup.py sdist bdist_wheel
+   ```
 
    Note: No need to add `--unversal` parameter, since `biothings` package now requires Python3, with no support of Python2 any more.
 
 5. Test the package built locally:
 
-    ```bash
-    pip install dist/biothings-0.9.0-py3-none-any.whl
-    ```
+   ```bash
+   pip install dist/biothings-0.9.0-py3-none-any.whl
+   ```
 
    And run any local test as needed (e.g. run nosetests on a local BioThings API instance).
 
 6. Prepare github repo for the release:
 
    * Create a version branch if not already (no need for every micro versions):
 
-        ```bash
-        git checkout -b 0.11.x
-        git push -u origin "0.11.x"
-        ```
+     ```bash
+     git checkout -b 0.11.x
+     git push -u origin "0.11.x"
+     ```
 
      Note: future version-specific bug-fixes (with increased micro versions) will go to this branch (possibly cherry-picked from `master` branch).
 
    * Create a tag for each released version (with "v" prefix):
 
-        ```bash
-        git tag -a "v0.9.0" -m "tagging v0.9.0 for release"
-        ```
+     ```bash
+     git tag -a "v0.9.0" -m "tagging v0.9.0 for release"
+     ```
 
    * If everything looks good, push to the remote:
 
-        ```bash
-        git push --tags
-        ```
+     ```bash
+     git push --tags
+     ```
 
 7. Publish a new release using Github Action
 
@@ -58,9 +60,9 @@
 
 8. Alternatively, upload manually a new release to PyPI:
 
-    ```bash
-    twine upload dist/*
-    ```
+   ```bash
+   twine upload dist/*
+   ```
 
     Note: make sure `dist` folder contains only the new versions you want to publish.