Update master branch

.flake8

@@ -11,4 +11,6 @@ exclude =
     venv,
     env,
     .venv,
+    ./venv-docs,
+    docs,
     .env

.github/workflows/bar-api.yml

@@ -13,7 +13,7 @@ jobs:
     runs-on: Ubuntu-20.04
     strategy:
       matrix:
-        python-version: [3.7, 3.8, 3.9, 3.10.0]
+        python-version: [3.7, 3.8, 3.9, 3.10.1]
 
     services:
       redis:
@@ -58,5 +58,4 @@ jobs:
     - name: "Upload coverage to Codecov"
       uses: codecov/codecov-action@v2
       with:
-        fail_ci_if_error: true
-        token: ${{ secrets.CODECOV_TOKEN }} # not required for public repos
+        fail_ci_if_error: false

.gitignore

@@ -11,6 +11,7 @@ __pycache__/
 build/
 develop-eggs/
 dist/
+docs/build/
 downloads/
 eggs/
 .eggs/
@@ -106,6 +107,7 @@ celerybeat.pid
 .venv
 env/
 venv/
+venv-docs/
 ENV/
 env.bak/
 venv.bak/
@@ -136,3 +138,6 @@ dmypy.json
 **/.DS_Store
 
 .vscode/
+output/*
+!output
+!output/placeholder.txt

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.10.0-bullseye
+FROM python:3.10.1-bullseye
 
 WORKDIR /usr/src/app
 

README.md

@@ -3,75 +3,6 @@
 **master**: [![Build Status](https://github.com/BioAnalyticResource/BAR_API/workflows/BAR-API/badge.svg?branch=master)](https://github.com/BioAnalyticResource/BAR_API/actions?query=branch%3Amaster) [![codecov](https://codecov.io/gh/BioAnalyticResource/BAR_API/branch/master/graph/badge.svg?token=QSYUWTRYEV)](https://codecov.io/gh/BioAnalyticResource/BAR_API) **dev**: [![Build Status](https://github.com/BioAnalyticResource/BAR_API/workflows/BAR-API/badge.svg?branch=dev)](https://github.com/BioAnalyticResource/BAR_API/actions?query=branch%3Adev) [![codecov](https://codecov.io/gh/BioAnalyticResource/BAR_API/branch/dev/graph/badge.svg?token=QSYUWTRYEV)](https://codecov.io/gh/BioAnalyticResource/BAR_API)
 
 [![Website Status](https://img.shields.io/website?url=http%3A%2F%2Fbar.utoronto.ca%2Fapi%2F)](http://bar.utoronto.ca/api/) ![GitHub repo size](https://img.shields.io/github/repo-size/BioAnalyticResource/BAR_API) [![LGTM Alerts](https://img.shields.io/lgtm/alerts/github/BioAnalyticResource/BAR_API)](https://lgtm.com/projects/g/BioAnalyticResource/BAR_API/?mode=list) [![LGTM
-Grade](https://img.shields.io/lgtm/grade/python/github/BioAnalyticResource/BAR_API)](https://lgtm.com/projects/g/BioAnalyticResource/BAR_API/latest/files/?sort=name&dir=ASC&mode=heatmap) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+Grade](https://img.shields.io/lgtm/grade/python/github/BioAnalyticResource/BAR_API)](https://lgtm.com/projects/g/BioAnalyticResource/BAR_API/latest/files/?sort=name&dir=ASC&mode=heatmap) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Documentation Status](https://readthedocs.org/projects/bar-api/badge/?version=latest)](https://bar-api.readthedocs.io/en/latest/?badge=latest)
 
-This is the official repository for the Bio-Analytic Resource API.
-
-## Status
-
-Apart from Travis CI and testing on the live BAR, we frequently test the API on systems that are not BAR for cross-platform compatibility. The most recent test is on the following systems:
-
-* OpenBSD 6.7-CURRENT (Maria DB 10.4.12v1, Python 3.8.3 and Redis 6.0.5)
-* FreeBSD 12.1-RELEASE-p6 (MySQL 8.0.20, Python 3.8.3 and Redis 5.0.9)
-
-## Run on your own computer with Docker
-
-1. Install [Docker](https://docs.docker.com/get-docker/)
-2. Install [Docker Compose](https://docs.docker.com/compose/install/)
-3. Install [Git](https://git-scm.com/downloads)
-4. Clone this repository and change directory to ```BAR_API```
-5. Build docker images
-```
-docker-compose build
-```
-6. Run docker containers (-d is detached)
-```
-docker-compose up -d
-```
-7. Load ```http://localhost:5000/``` in a web browser. Enjoy :)
-
-## Run on your own computer without Docker
-
-1. Install [MySQL](https://www.mysql.com/products/community/) or [Maria DB](https://mariadb.com/downloads/).
-2. Install [Redis](https://redis.io/download).
-3. Install [Python](https://www.python.org/downloads/) or [Pypy](https://www.pypy.org/download.html). Note: Python 2 is not supported.
-4. Install [Git](https://git-scm.com/downloads)
-5. (Optional) On Debian based systems, you may also need to install ```libmysqlclient-dev``` and ```python3-dev```. On FreeBSD, you may need to install ```py38-sqlite3```. We will update this step as we come across more OS dependencies. 
-6. Clone this repository and change directory to ```BAR_API```
-7. Set up a virtual environment
-```
-python3 -m venv venv
-```
-8. Activate the virtual environment. Bash/Zsh:
-```
-source venv/bin/activate
-```
-csh/tcsh:
-```
-source venv/bin/activate.csh
-```
-9. Install requirements
-```
-pip install --upgrade pip setuptools wheel
-pip install -r requirements.txt
-```
-10. Copy ```config/BAR_API.cgi``` to your preferred directory, for example:
-```
-cp config/BAR_API.cgi ~/.config/
-```
-Add, update, and modify passwords and environment variables as needed.
-
-11. Copy ```./config/init.sh``` to BAR_API directory:
-```
-cp config/init.sh .
-```
-Change passwords in ```./init.sh``` and run this script to load the databases:
-```
-./init.sh
-```
-Then delete ```./init.sh```
-
-12. Edit ```./api/__init__.py``` and update the location of your BAR_API.cfg file if you have changed it.
-13. Run ```pytest```. Tests should pass if the system is set up correctly.
-14. Run ```python app.py``` to start.
-15. Load ```http://localhost:5000/``` in a web browser. Enjoy :)
+This is the official repository for the Bio-Analytic Resource API. The API documentation can be found [here](https://bar-api.readthedocs.io/en/latest/).

api/__init__.py

@@ -20,14 +20,28 @@ def create_app():
         print("We are now loading configuration.")
         bar_app.config.from_pyfile(os.getcwd() + "/config/BAR_API.cfg", silent=True)
         if bar_app.config.get("ADMIN_ENCRYPT_KEY"):
-            os.environ["ADMIN_ENCRYPT_KEY"] = bar_app.config.get("ADMIN_ENCRYPT_KEY")
+            os.environ["ADMIN_ENCRYPT_KEY"] = bar_app.config.get(
+                "TEST_ADMIN_ENCRYPT_KEY"
+            )
         if bar_app.config.get("ADMIN_PASSWORD_FILE"):
             os.environ["ADMIN_PASSWORD_FILE"] = bar_app.config.get(
-                "ADMIN_PASSWORD_FILE"
+                "TEST_ADMIN_PASSWORD_FILE"
             )
+        if bar_app.config.get("ADMIN_EMAIL"):
+            os.environ["ADMIN_EMAIL"] = bar_app.config.get("ADMIN_EMAIL")
+        if bar_app.config.get("EMAIL_PASS_KEY"):
+            os.environ["EMAIL_PASS_KEY"] = bar_app.config.get("EMAIL_PASS_KEY")
+        if bar_app.config.get("EMAIL_PASS_FILE"):
+            os.environ["EMAIL_PASS_FILE"] = bar_app.config.get("EMAIL_PASS_FILE")
     elif os.environ.get("BAR"):
         # The BAR
         bar_app.config.from_pyfile(os.environ.get("BAR_API_PATH"), silent=True)
+        if bar_app.config.get("ADMIN_EMAIL"):
+            os.environ["ADMIN_EMAIL"] = bar_app.config.get("ADMIN_EMAIL")
+        if bar_app.config.get("EMAIL_PASS_KEY"):
+            os.environ["EMAIL_PASS_KEY"] = bar_app.config.get("EMAIL_PASS_KEY")
+        if bar_app.config.get("EMAIL_PASS_FILE"):
+            os.environ["EMAIL_PASS_FILE"] = bar_app.config.get("EMAIL_PASS_FILE")
         if bar_app.config.get("ADMIN_ENCRYPT_KEY"):
             os.environ["ADMIN_ENCRYPT_KEY"] = bar_app.config.get("ADMIN_ENCRYPT_KEY")
         if bar_app.config.get("ADMIN_PASSWORD_FILE"):
@@ -51,6 +65,12 @@ def create_app():
             os.environ["ADMIN_PASSWORD_FILE"] = bar_app.config.get(
                 "ADMIN_PASSWORD_FILE"
             )
+        if bar_app.config.get("ADMIN_EMAIL"):
+            os.environ["ADMIN_EMAIL"] = bar_app.config.get("ADMIN_EMAIL")
+        if bar_app.config.get("EMAIL_PASS_KEY"):
+            os.environ["EMAIL_PASS_KEY"] = bar_app.config.get("EMAIL_PASS_KEY")
+        if bar_app.config.get("EMAIL_PASS_FILE"):
+            os.environ["EMAIL_PASS_FILE"] = bar_app.config.get("EMAIL_PASS_FILE")
         if bar_app.config.get("DRIVE_LIST_KEY"):
             os.environ["DRIVE_LIST_KEY"] = bar_app.config.get("DRIVE_LIST_KEY")
         if bar_app.config.get("DRIVE_LIST_FILE"):
@@ -103,6 +123,7 @@ def create_app():
     from api.resources.gene_annotation import gene_annotation
     from api.resources.interactions import itrns
     from api.resources.gene_localizations import loc
+    from api.resources.efp_image import efp_image
 
     bar_api.add_namespace(gene_information)
     bar_api.add_namespace(rnaseq_gene_expression)
@@ -115,6 +136,7 @@ def create_app():
     bar_api.add_namespace(gene_annotation)
     bar_api.add_namespace(itrns)
     bar_api.add_namespace(loc)
+    bar_api.add_namespace(efp_image)
     bar_api.init_app(bar_app)
     return bar_app
 

api/resources/api_manager.py

@@ -5,11 +5,16 @@
 from flask_restx import Namespace, Resource
 from datetime import datetime
 from sqlalchemy.exc import SQLAlchemyError
+from sqlalchemy.types import String, Float
 import os
 import uuid
 import requests
 import pandas
 from cryptography.fernet import Fernet
+from smtplib import SMTP_SSL
+from ssl import create_default_context
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
 
 CAPTCHA_KEY_FILE = "/home/bpereira/data/bar.summarization/key"
 
@@ -32,6 +37,56 @@ def check_admin_pass(password):
         else:
             return False
 
+    @staticmethod
+    def validate_captcha(value):
+        """Validates a reCaptcha value using our secret token"""
+        if os.environ.get("BAR"):
+            with open(CAPTCHA_KEY_FILE, "rb") as f:
+                for line in f:
+                    key = line
+            if key:
+                ret = requests.post(
+                    "https://www.google.com/recaptcha/api/siteverify",
+                    data={"secret": key, "response": value},
+                )
+                return ret.json()["success"]
+            else:
+                return False
+        else:
+            return True
+
+    @staticmethod
+    def send_email_notification():
+        if os.environ.get("BAR"):
+            with open(os.environ.get("ADMIN_EMAIL"), "r") as f:
+                for line in f:
+                    recipient = line
+            port = 465
+            key = os.environ.get("EMAIL_PASS_KEY")
+            cipher_suite = Fernet(key)
+            with open(os.environ.get("EMAIL_PASS_FILE"), "rb") as f:
+                for line in f:
+                    encrypted_key = line
+            uncipher_text = cipher_suite.decrypt(encrypted_key)
+            password = bytes(uncipher_text).decode("utf-8")
+            context = create_default_context()
+            smtp_server = "smtp.gmail.com"
+            sender_email = "bar.summarization@gmail.com"
+            subject = "[Bio-Analytic Resource] New API key request"
+            text = """\
+                There is a new API key request.
+                You can approve or reject it at http://bar.utoronto.ca/~bpereira/webservices/bar-request-manager/build/index.html
+            """
+            m_text = MIMEText(text, _subtype="plain", _charset="UTF-8")
+            msg = MIMEMultipart()
+            msg["From"] = sender_email
+            msg["To"] = recipient
+            msg["Subject"] = subject
+            msg.attach(m_text)
+            with SMTP_SSL(smtp_server, port, context=context) as server:
+                server.login("bar.summarization@gmail.com", password)
+                server.sendmail(sender_email, recipient, msg.as_string())
+
 
 @api_manager.route("/validate_admin_password", methods=["POST"], doc=False)
 class ApiManagerValidate(Resource):
@@ -72,22 +127,27 @@ def post(self):
 class ApiManagerRequest(Resource):
     def post(self):
         if request.method == "POST":
-            response_json = request.get_json()
-            df = pandas.DataFrame.from_records([response_json])
-            con = db.get_engine(bind="summarization")
-            try:
-                reqs = Requests()
-                users = Users()
-                row_req = reqs.query.filter_by(email=df.email[0]).first()
-                row_users = users.query.filter_by(email=df.email[0]).first()
-
-                if row_req is None and row_users is None:
-                    df.to_sql("requests", con, if_exists="append", index=False)
-                    return BARUtils.success_exit("Data added")
-                else:
-                    return BARUtils.error_exit("E-mail already in use"), 409
-            except SQLAlchemyError:
-                return BARUtils.error_exit("Internal server error"), 500
+            captchaVal = request.headers.get("captchaVal")
+            if ApiManagerUtils.validate_captcha(captchaVal):
+                response_json = request.get_json()
+                df = pandas.DataFrame.from_records([response_json])
+                con = db.get_engine(bind="summarization")
+                try:
+                    reqs = Requests()
+                    users = Users()
+                    row_req = reqs.query.filter_by(email=df.email[0]).first()
+                    row_users = users.query.filter_by(email=df.email[0]).first()
+
+                    if row_req is None and row_users is None:
+                        df.to_sql("requests", con, if_exists="append", index=False)
+                        ApiManagerUtils.send_email_notification()
+                        return BARUtils.success_exit("Data added")
+                    else:
+                        return BARUtils.error_exit("E-mail already in use"), 409
+                except SQLAlchemyError:
+                    return BARUtils.error_exit("Internal server error"), 500
+            else:
+                return BARUtils.error_exit("Failed Captcha verification")
 
 
 @api_manager.route("/get_pending_requests", methods=["POST"], doc=False)
@@ -167,37 +227,34 @@ def post(self):
                             "date_added": datetime.now(),
                             "status": "user",
                             "api_key": key,
-                            "uses_left": 25,
+                            "uses_left": 100,
                         }
                     )
                     for row in rows
                 ]
                 df = pandas.DataFrame.from_records([values[0]])
+                values_df = pandas.DataFrame(columns=["Gene", "Sample", "Value"])
                 con = db.get_engine(bind="summarization")
                 try:
                     df.to_sql("users", con, if_exists="append", index=False)
+                    values_df.to_sql(
+                        key,
+                        con,
+                        index_label="index",
+                        dtype={
+                            values_df.index.name: String(42),
+                            "Gene": String(32),
+                            "Sample": String(32),
+                            "Value": Float,
+                        },
+                        if_exists="append",
+                        index=True,
+                    )
                     el = table.query.filter_by(email=email).one()
                     db.session.delete(el)
+                    db.session.commit()
                 except SQLAlchemyError:
                     return BARUtils.error_exit("Internal server error"), 500
                 return BARUtils.success_exit(key)
             else:
                 return BARUtils.error_exit("Forbidden"), 403
-
-
-@api_manager.route("/validate_captcha", methods=["POST"], doc=False)
-class ApiManagerCaptchaValidate(Resource):
-    def post(self):
-        """Validates a reCaptcha value using our secret token"""
-        if request.method == "POST":
-            json = request.get_json()
-            value = json["response"]
-            key = os.environ.get("CAPTCHA_KEY_FILE")
-            if key:
-                ret = requests.post(
-                    "https://www.google.com/recaptcha/api/siteverify",
-                    data={"secret": key, "response": value},
-                )
-                return BARUtils.success_exit(ret.text)
-            else:
-                return BARUtils.error_exit("Forbidden: CAPTCHA key is not found"), 403