Add linter hooks and update the requirements as well as the readme. (#62

) * Add linter hooks and update the requirements as well as the readme. * Fix Circle CI. * Fix the ci build. * Fix the ci build. * Format and fix the styles. (#63) * Fix the wildcard linting error. * Fix the config of flake8 so it won't check the site-packages in venv dir.
HumanCellAtlas · Apr 8, 2019 · 65790bb · 65790bb
1 parent 415b811
commit 65790bb
Show file tree

Hide file tree

Showing 32 changed files with 2,320 additions and 1,000 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -42,6 +42,15 @@ jobs:
       # run tests!
       # https://pytest.org
       # And upload reports to codecov.io
+      - run:
+          name: linting test
+          command: |
+            . venv/bin/activate
+            # Check Black code style compliance
+            black ./ --skip-string-normalization --check --exclude venv
+            # Check PEP-8 compliance
+            flake8
+
       - run:
           name: run tests
           command: |
@@ -53,4 +62,3 @@ jobs:
       - store_artifacts:
           path: test-reports
           destination: test-reports
-
diff --git a/.flake8 b/.flake8
@@ -0,0 +1,20 @@
+# We ignore the following PEP-8 styles:
+
+# E203  whitespace before ‘:’
+# E266  too many leading ‘#’ for block comment
+# E501  line too long (82 > 79 characters) (^)
+# W503  line break occurred before a binary operator
+# F841  local variable is assigned to but never used
+# W605  invalid escape sequence (causes false alarms around regex)
+
+# Note: (^) These checks can be disabled at the
+# line level using the # noqa special comment.
+# This possibility should be reserved for special cases.
+
+[flake8]
+ignore = E203, E266, E501, W503, F841, W605
+max-complexity = 18
+select = B,C,E,F,W,T4,B9
+exclude = 
+    # No need to traverse the virtualenv directory, which'll be created by Circle CI
+    venv
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,17 @@
+repos:
+-   repo: https://github.com/ambv/black
+    rev: 19.3b0
+    hooks:
+    - id: black
+      language_version: python3.6
+      # Using args here is not recommended by Black:
+      # https://black.readthedocs.io/en/stable/version_control_integration.html
+      # But since we only have one argument here, and
+      # we don't force developers to use editor plugins,
+      # putting the args here seems to be fine
+      args: [./, --skip-string-normalization]
+
+-   repo: https://gitlab.com/pycqa/flake8
+    rev: 3.7.7
+    hooks:
+    -   id: flake8
diff --git a/README.rst b/README.rst
@@ -1,25 +1,43 @@
 Single Cell Tools
-=======
+#################
 
-.. image:: https://circleci.com/gh/HumanCellAtlas/sctools/tree/master.svg?style=shield
-    :target: https://circleci.com/gh/HumanCellAtlas/sctools/tree/master
+.. image:: https://img.shields.io/circleci/project/github/HumanCellAtlas/sctools.svg?label=Unit%20Test%20on%20Circle%20CI%20&style=flat-square&logo=circleci
+  :target: https://circleci.com/gh/HumanCellAtlas/sctools/tree/master
+  :alt: Unit Test Status
 
-.. image:: https://codecov.io/gh/HumanCellAtlas/sctools/branch/master/graph/badge.svg
+.. image:: https://img.shields.io/codecov/c/github/HumanCellAtlas/sctools/master.svg?label=Test%20Coverage&logo=codecov&style=flat-square
   :target: https://codecov.io/gh/HumanCellAtlas/sctools
+  :alt: Test Coverage on Codecov
 
-.. image:: https://readthedocs.org/projects/sctools/badge/?version=latest
-  :target: https://sctools.readthedocs.io/en/latest/?badge=latest
+.. image:: https://img.shields.io/readthedocs/sctools/latest.svg?label=ReadtheDocs%3A%20Latest&logo=Read%20the%20Docs&style=flat-square
+  :target: http://sctools.readthedocs.io/en/latest/?badge=latest
   :alt: Documentation Status
 
 .. image:: https://img.shields.io/snyk/vulnerabilities/github/HumanCellAtlas/sctools/requirements.txt.svg?label=Snyk%20Vulnerabilities&logo=Snyk
   :target: https://snyk.io/test/github/HumanCellAtlas/sctools/?targetFile=requirements.txt
   :alt: Snyk Vulnerabilities for GitHub Repo (Specific Manifest)
 
+.. image:: https://img.shields.io/github/release/HumanCellAtlas/sctools.svg?label=Latest%20Release&style=flat-square&colorB=green
+  :target: https://github.com/HumanCellAtlas/sctools/releases
+  :alt: Latest Release
+
+.. image:: https://img.shields.io/github/license/HumanCellAtlas/sctools.svg?style=flat-square
+  :target: https://img.shields.io/github/license/HumanCellAtlas/sctools.svg?style=flat-square
+  :alt: License
+
+.. image:: https://img.shields.io/badge/python-3.6-green.svg?style=flat-square&logo=python&colorB=blue
+  :target: https://img.shields.io/badge/python-3.6-green.svg?style=flat-square&logo=python&colorB=blue
+  :alt: Language
+
+.. image:: https://img.shields.io/badge/Code%20Style-black-000000.svg?style=flat-square
+  :target: https://github.com/ambv/black
+  :alt: Code Style
+
 Single Cell Tools provides utilities for manipulating sequence data formats suitable for use in
 distributed systems analyzing large biological datasets.
 
 Download and Installation
--------------------------
+=========================
 
 .. code bash
    git clone https://github.com/humancellatlas/sctools.git
@@ -28,13 +46,13 @@ Download and Installation
    pytest  # verify installation; run tests
 
 sctools Package
----------------
+===============
 
 The sctools package provides both command line utilities and classes designed for use in python
 programs.
 
 Command Line Utilities
-----------------------
+======================
 
 1. Attach10XBarcodes: Attached barcodes stored in fastq files to reads in an unaligned bam file
 2. SplitBam: Split a bam file into chunks, guaranteeing that cells are contained in 1 chunk
@@ -44,7 +62,7 @@ Command Line Utilities
 6. MergeCellMetrics Merge cell metrics calculated from different chunks of an experiment
 
 Main Package Classes
---------------------
+====================
 
 1. **Platform**: an abstract class that defines a common data structure for different 3' sequencing
    formats. All algorithms and methods in this package that are designed to work on 3' sequencing data
@@ -74,7 +92,7 @@ Main Package Classes
 
 
 Viewing Test Results and Coverage
----------------------------------
+=================================
 
 To calculate and view test coverage cd to the ``sctools`` directory and
 type the following two commands to generate the report and open it in your web browser:
@@ -85,7 +103,7 @@ type the following two commands to generate the report and open it in your web b
    open cov_html/index.html
 
 Definitions
------------
+===========
 
 Several definitions are helpful to understand how sequence data is analyzed.
 
@@ -118,3 +136,22 @@ position it aligns to on that chromosome, after correcting for clipping that the
 7. **Bam/Sam file**: The GA4GH standard file type for the storage of aligned sequencing reads.
 Unless specified, our Single Cell Tools will operate over bam files containing either aligned or
 unaligned reads
+
+Development
+===========
+
+Code Style
+----------
+The sctools code base is complying with the PEP-8 and using `Black <https://github.com/ambv/black>`_ to
+format our code, in order to avoid "nitpicky" comments during the code review process so we spend more time discussing about the logic, 
+not code styles.
+
+In order to enable the auto-formatting in the development process, you have to spend a few seconds setting 
+up the ``pre-commit`` the first time you clone the repo:
+
+1. Install ``pre-commit`` by running: ``pip install pre-commit`` (or simply run ``pip install -r requirements.txt``).
+2. Run `pre-commit install` to install the git hook.
+
+Once you successfully install the ``pre-commit`` hook to this repo, the Black linter/formatter will be automatically triggered and run on this repo. Please make sure you followed the above steps, otherwise your commits might fail at the linting test!
+
+If you really want to manually trigger the linters and formatters on your code, make sure ``Black`` and ``flake8`` are installed in your Python environment and run ``flake8 DIR1 DIR2`` and ``black DIR1 DIR2 --skip-string-normalization`` respectively.
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -45,7 +45,7 @@
     'sphinx.ext.mathjax',
     'sphinx.ext.viewcode',
     'sphinx.ext.napoleon',
-    'sphinx.ext.autosummary'
+    'sphinx.ext.autosummary',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -117,15 +117,12 @@
     # The paper size ('letterpaper' or 'a4paper').
     #
     # 'papersize': 'letterpaper',
-
     # The font size ('10pt', '11pt' or '12pt').
     #
     # 'pointsize': '10pt',
-
     # Additional stuff for the LaTeX preamble.
     #
     # 'preamble': '',
-
     # Latex figure (float) alignment
     #
     # 'figure_align': 'htbp',
@@ -135,19 +132,15 @@
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'SCTools.tex', 'SC Tools Documentation',
-     'Ambrose J. Carr', 'manual'),
+    (master_doc, 'SCTools.tex', 'SC Tools Documentation', 'Ambrose J. Carr', 'manual')
 ]
 
 
 # -- Options for manual page output ------------------------------------------
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [
-    (master_doc, 'sctools', 'SC Tools Documentation',
-     [author], 1)
-]
+man_pages = [(master_doc, 'sctools', 'SC Tools Documentation', [author], 1)]
 
 
 # -- Options for Texinfo output ----------------------------------------------
@@ -156,9 +149,15 @@
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, 'SCTools', 'SC Tools Documentation',
-     author, 'SCTools', 'One line description of project.',
-     'Miscellaneous'),
+    (
+        master_doc,
+        'SCTools',
+        'SC Tools Documentation',
+        author,
+        'SCTools',
+        'One line description of project.',
+        'Miscellaneous',
+    )
 ]
 
 

diff --git a/requirements.txt b/requirements.txt
@@ -8,3 +8,6 @@ scipy>=1.1.0
 crimson>=0.3.0
 setuptools>=40.4.3
 setuptools_scm>=3.1.0
+black==19.3b0
+flake8==3.7.7
+pre-commit==1.14.4
diff --git a/setup.py b/setup.py
@@ -33,23 +33,22 @@
         'setuptools>=40.4.3',
         'scipy>=1.0.0',
         'crimson>=0.3.0',
-
     ],
     entry_points={
-            'console_scripts': [
-                'AttachBarcodes = sctools.platform:BarcodePlatform.attach_barcodes',
-                'Attach10xBarcodes = sctools.platform:TenXV2.attach_barcodes',
-                'SplitBam = sctools.platform:GenericPlatform.split_bam',
-                'CalculateGeneMetrics = sctools.platform:GenericPlatform.calculate_gene_metrics',
-                'CalculateCellMetrics = sctools.platform:GenericPlatform.calculate_cell_metrics',
-                'MergeGeneMetrics = sctools.platform:GenericPlatform.merge_gene_metrics',
-                'MergeCellMetrics = sctools.platform:GenericPlatform.merge_cell_metrics',
-                'CreateCountMatrix = sctools.platform:GenericPlatform.bam_to_count_matrix',
-                'MergeCountMatrices = sctools.platform:GenericPlatform.merge_count_matrices',
-                'TagSortBam = sctools.platform:GenericPlatform.tag_sort_bam',
-                'VerifyBamSort = sctools.platform:GenericPlatform.verify_bam_sort',
-                'GroupQCs = sctools.platform:GenericPlatform.group_qc_outputs'
-            ]
+        'console_scripts': [
+            'AttachBarcodes = sctools.platform:BarcodePlatform.attach_barcodes',
+            'Attach10xBarcodes = sctools.platform:TenXV2.attach_barcodes',
+            'SplitBam = sctools.platform:GenericPlatform.split_bam',
+            'CalculateGeneMetrics = sctools.platform:GenericPlatform.calculate_gene_metrics',
+            'CalculateCellMetrics = sctools.platform:GenericPlatform.calculate_cell_metrics',
+            'MergeGeneMetrics = sctools.platform:GenericPlatform.merge_gene_metrics',
+            'MergeCellMetrics = sctools.platform:GenericPlatform.merge_cell_metrics',
+            'CreateCountMatrix = sctools.platform:GenericPlatform.bam_to_count_matrix',
+            'MergeCountMatrices = sctools.platform:GenericPlatform.merge_count_matrices',
+            'TagSortBam = sctools.platform:GenericPlatform.tag_sort_bam',
+            'VerifyBamSort = sctools.platform:GenericPlatform.verify_bam_sort',
+            'GroupQCs = sctools.platform:GenericPlatform.group_qc_outputs',
+        ]
     },
     classifiers=CLASSIFIERS,
     include_package_data=True,

diff --git a/src/sctools/__init__.py b/src/sctools/__init__.py
@@ -1,3 +1,4 @@
+# flake8: noqa
 from . import bam
 from . import encodings
 from . import barcode