make: ci recipe fails at setup

[dprelipcean]

What happens

I wanted to run in dev mode, hence I used

$ CLUSTER_CONFIG=dev make ci

The recipe went through without problems; however, when trying to set up my environment:

$ eval $(reana-dev setup-environment)
ERROR:  relation "user_" does not exist
LINE 1: SELECT access_token FROM user_
                                 ^
command terminated with exit code 1
Environment variables could not be generated: 
Command '['kubectl', 'exec', '-t', 'db-7df68b647-vnb4v', '--', 'psql', '-U', 'reana', 'reana', '-c', 'SELECT access_token FROM user_']' returned non-zero exit status 1.

Here is where the error is produced.

When running it normally, i.e. simply make ci, everything works as expected.

More info

Looking at my pods, the server pod actually fails:

$ kubectl get pods
NAME                                                   READY   STATUS             RESTARTS   AGE
cache-f74dd5fd8-jzvpk                                  1/1     Running            0          86m
db-7df68b647-7jj4h                                     1/1     Running            0          86m
message-broker-6758d6f768-sd76w                        1/1     Running            0          86m
my-shell-6d9885f5df-l2pww                              1/1     Running            1          168m
server-59bdf5b75b-4g26v                                0/2     CrashLoopBackOff   42         86m
wdb-6c44658878-cmz4q                                   1/1     Running            0          86m
workflow-controller-bf974dd-dq6fj                      2/2     Running            0          86m

Looking at the logs:

$ kubectl logs server-59bdf5b75b-4g26v  server
Creating REANA database ...
/usr/local/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py:835: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '

/usr/local/lib/python3.6/site-packages/invenio_config/default.py:74: UserWarning: Set configuration variable SECRET_KEY with random string
  UserWarning)

/usr/local/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py:835: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '

Usage: invenio [OPTIONS] COMMAND [ARGS]...
Try "invenio --help" for help.

Error: No such command "reana-db".

and

$ kubectl logs server-59bdf5b75b-4g26v  scheduler
/usr/local/lib/python3.6/site-packages/flask_sqlalchemy/__init__.py:835: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
2019-09-06 11:41:52,037 | passlib.registry | MainThread | DEBUG | registered 'invenio_aes_encrypted_email' handler: <class 'invenio_accounts.hash.InvenioAesEncryptedEmail'>
2019-09-06 11:41:52,071 | passlib.registry | MainThread | DEBUG | registered 'pbkdf2_sha512' handler: <class 'passlib.handlers.pbkdf2.pbkdf2_sha512'>
2019-09-06 11:41:52,075 | passlib.registry | MainThread | DEBUG | registered 'sha256_crypt' handler: <class 'passlib.handlers.sha2_crypt.sha256_crypt'>
2019-09-06 11:41:52,078 | passlib.registry | MainThread | DEBUG | registered 'hex_md5' handler: <class 'passlib.handlers.digests.hex_md5'>
Usage: flask [OPTIONS] COMMAND [ARGS]...
Try "flask --help" for help.

Error: No such command "start-scheduler".

with

$ kubectl describe pod server-59bdf5b75b-4g26v
Name:           server-59bdf5b75b-4g26v
Namespace:      default
Priority:       0
Node:           minikube/192.168.122.240
Start Time:     Fri, 06 Sep 2019 12:17:46 +0200
Labels:         app=server
                pod-template-hash=59bdf5b75b
Annotations:    <none>
Status:         Running
IP:             172.17.0.7
Controlled By:  ReplicaSet/server-59bdf5b75b
Containers:
  server:
    Container ID:  docker://b39de5b315b452350f461cbebc225194d129c770a23c2322c24fe252abff357a
    Image:         reanahub/reana-server:latest
    Image ID:      docker://sha256:0fbc333f172f1f3e32d9ca24e236adf1dd154968cee78ce81708641b9f2733cd
    Port:          5000/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
      -c
    Args:
      set -e; ./scripts/setup; invenio run -h 0.0.0.0 -p 5000
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Fri, 06 Sep 2019 13:41:48 +0200
      Finished:     Fri, 06 Sep 2019 13:41:53 +0200
    Ready:          False
    Restart Count:  21
    Environment:
      REANA_DB_PORT:             5432
      REANA_DB_HOST:             db
      REANA_DB_PASSWORD:         reana
      REANA_DB_USERNAME:         reana
      REANA_DB_NAME:             reana
      SHARED_VOLUME_PATH:        /var/reana
      WDB_SOCKET_SERVER:         wdb
      WDB_NO_BROWSER_AUTO_OPEN:  True
      FLASK_ENV:                 development
    Mounts:
      /code from reana-server-code (rw)
      /var/reana from reana-shared-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qhv2v (ro)
  scheduler:
    Container ID:  docker://47837302dc26e529e9806abd5106052beab01247a11f941560bc3f3630616f82
    Image:         reanahub/reana-server:latest
    Image ID:      docker://sha256:0fbc333f172f1f3e32d9ca24e236adf1dd154968cee78ce81708641b9f2733cd
    Port:          <none>
    Host Port:     <none>
    Command:
      flask
      start-scheduler
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Fri, 06 Sep 2019 13:41:48 +0200
      Finished:     Fri, 06 Sep 2019 13:41:53 +0200
    Ready:          False
    Restart Count:  21
    Environment:
      REANA_DB_PORT:             5432
      REANA_DB_HOST:             db
      REANA_DB_PASSWORD:         reana
      REANA_DB_USERNAME:         reana
      REANA_DB_NAME:             reana
      SHARED_VOLUME_PATH:        /var/reana
      WDB_SOCKET_SERVER:         wdb
      WDB_NO_BROWSER_AUTO_OPEN:  True
      FLASK_ENV:                 development
    Mounts:
      /code from reana-server-code (rw)
      /var/reana from reana-shared-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qhv2v (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  reana-shared-volume:
    Type:          HostPath (bare host directory volume)
    Path:          /var/reana
    HostPathType:  
  reana-server-code:
    Type:          HostPath (bare host directory volume)
    Path:          /code/reana-server
    HostPathType:  
  default-token-qhv2v:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qhv2v
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                    From               Message
  ----     ------   ----                   ----               -------
  Warning  BackOff  6m53s (x381 over 86m)  kubelet, minikube  Back-off restarting failed container
  Warning  BackOff  108s (x406 over 86m)   kubelet, minikube  Back-off restarting failed container

@diegodelemos I see that invenio has something to deal with this, maybe it's related to the recent updates?

[diegodelemos]

tl;dr

Can you try with reana-dev git-clean -c CLUSTER and run it again?

Explanation

This sounds like the problem described in reanahub/reana-workflow-controller#64, to which Tibor proposed a solution (reanahub/reana-workflow-controller#64 (comment)), and later was integrated into reana-dev command as reana-dev git-clean, see the docs https://reana.readthedocs.io/en/latest/developerguide.html.

[diegodelemos]

You will need to install the reana-server package in your local machine:

(reana-server) ~/src/reana-server $ pip install -e .
...

The reason is that the editable mode creates and uses the egg-info (therefore expects it to exist) to save you from reinstalling the application after every code change. Then what happens is that the .eggs-info is not there, because REANA-Server was never installed in your computer or because you run git clean ... and when the application looks for the entrypoint to resolve the reana-db command, the reana-server.eggs-info/entry_points.txt is not there.

[tiborsimko]

N.B. The fix was merged in #198.