Fix `bin/e-mission-py.bash` in e-mission-docker to work with the new activation script #619

xubowenhaoren · 2021-02-22T04:43:25Z

Hello,

I am trying to deploy a new instance of the em-server e-mission server docker. After installing docker and docker-compose on a Ubuntu 16.04 LTS server, I ran

git clone https://github.com/e-mission/e-mission-docker.git
cd e-mission-docker/
docker-compose -f examples/em-server/docker-compose.yml up -d

I then got the following output:

WARNING: Some services (db, web-server) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker sCreating emserver_db_1         ... done
Creating emserver_db_1         ... 
Creating emserver_web-server_1 ... done

However, after opening the docker shell and trying to load the test data, I got

root@7cfbc709fd74:/usr/src/app/e-mission-server# ./e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py emission/tests/data/real_examples/shankari_2015-07-22 test_july_22
./e-mission-py.bash: line 8: python: command not found

I noticed that you wrote about another method of running scripts outside of the docker shell here. I tried that but with no success:

root@valued-byte-6:~/e-mission-docker# CONTAINER=emserver_web-server_1 bin/e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py emission/tests/data/real_examples/shankari_2015-07-22 test_july_22
ARGS=bin/debug/load_timeline_for_day_and_user.py emission/tests/data/real_examples/shankari_2015-07-22 test_july_22
bash: activate: No such file or directory

The text was updated successfully, but these errors were encountered:

shankari · 2021-02-22T05:45:35Z

However, after opening the docker shell and trying to load the test data, I got

run source setup/activate.sh first. Once you see the (emission) environment activated, running commands in the container should work

I noticed that you wrote about another method of running scripts outside of the docker shell here. I tried that but with no success:

True, that doesn't work after the switch to the latest version of conda. Changing the title to reflect this.

xubowenhaoren · 2021-02-22T07:21:50Z

Thank you! I was able to get the test data intake pipeline running with the following steps:

root@valued-byte-6:~/e-mission-docker# docker ps -a
CONTAINER ID   IMAGE                              COMMAND                  CREATED       STATUS       PORTS                      NAMES
7cfbc709fd74   emission/e-mission-server:latest   "/bin/bash /start_sc…"   2 hours ago   Up 2 hours   0.0.0.0:8080->8080/tcp     emserver_web-server_1
749d2dacb094   mongo:3.4                          "docker-entrypoint.s…"   2 hours ago   Up 2 hours   0.0.0.0:27017->27017/tcp   emserver_db_1
root@valued-byte-6:~/e-mission-docker# docker exec -it 7cfbc709fd74 bash
root@7cfbc709fd74:/usr/src/app# cd e-mission-server/
root@7cfbc709fd74:/usr/src/app/e-mission-server# source setup/activate.sh
(emission) root@7cfbc709fd74:/usr/src/app/e-mission-server# ./e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py emission/tests/data/real_examples/shankari_2015-07-22 test_july_22

Then I got the following output:

Connecting to database URL db
emission/tests/data/real_examples/shankari_2015-07-22
Loading file emission/tests/data/real_examples/shankari_2015-07-22
After registration, test_july_22 -> e549851f-676b-4eee-ba8a-c11cc987653b
Finished loading 0 entries into the usercache and 1906 entries into the timeseries
(emission) root@7cfbc709fd74:/usr/src/app/e-mission-server# ./e-mission-py.bash bin/debug/intake_single_user.py -e test_july_22
Connecting to database URL db
analysis.debug.conf.json not configured, falling back to sample, default configuration
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/usr/src/app/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
2021-02-22T06:39:44.825951+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: moving to long term**********
2021-02-22T06:39:45.025620+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: updating incoming user inputs**********
2021-02-22T06:39:45.107904+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: filter accuracy if needed**********
2021-02-22T06:39:45.117240+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: segmenting into trips**********
2021-02-22T06:39:56.695637+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: segmenting into sections**********
2021-02-22T06:39:58.008523+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: smoothing sections**********
2021-02-22T06:39:59.677102+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: cleaning and resampling timeline**********
2021-02-22T06:40:04.013405+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: inferring transportation mode**********
Error while inferring modes, timestamp is unchanged
Traceback (most recent call last):
  File "/usr/src/app/e-mission-server/emission/analysis/classification/inference/mode/pipeline.py", line 41, in predict_mode
    mip.runPredictionPipeline(user_id, time_query)
  File "/usr/src/app/e-mission-server/emission/analysis/classification/inference/mode/pipeline.py", line 139, in runPredictionPipeline
    self.loadModelStage()
  File "/usr/src/app/e-mission-server/emission/analysis/classification/inference/mode/pipeline.py", line 156, in loadModelStage
    self.model = seedp.ModeInferencePipelineMovesFormat.loadModel()
  File "/usr/src/app/e-mission-server/emission/analysis/classification/inference/mode/seed/pipeline.py", line 93, in loadModel
    fd = open(SAVED_MODEL_FILENAME, "r")
FileNotFoundError: [Errno 2] No such file or directory: 'seed_model.json'
2021-02-22T06:40:04.119796+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: creating confirmed objects **********
2021-02-22T06:40:04.213817+00:00**********UUID e549851f-676b-4eee-ba8a-c11cc987653b: storing views to cache**********

I don't need the game functions for my project, so I ignored the missing habitica.json for now. But for seed_model.json, I found my previous issue and modified the multi-tier docker-compose file:

version: "3"
services:
  web-server:
    build: webapp
    depends_on:
      - db
    environment:
      - DB_HOST=db
      - WEB_SERVER_HOST=0.0.0.0
      - SERVER_BRANCH=gis-based-mode-detection
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    ports:
      # If you are exposing this directly, use 80:8080 or 443:8080
      # If you are using a reverse proxy, use a unique port and configure the
      # RP appropriately
      - "80:8080"
    networks:
      - emission
  analysis-server:
    build: analysis
    depends_on:
      - db
    environment:
      - DB_HOST=db
      - WEB_SERVER_HOST=0.0.0.0
      - SERVER_BRANCH=gis-based-mode-detection
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    networks:
      - emission
  db:
    image: mongo:4.4.0
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

    #Volumes is the preferred way to persist data generated by a container. In this case we use a volume to persist the contents
    #of the data base. Learn more about how to use volumes here: https://docs.docker.com/storage/volumes/
    # And learn how to configure volumes in your compose file here: https://docs.docker.com/compose/compose-file/#volume-configuration-reference
    volumes:
      - mongo-data:/data/db
    networks:
       - emission

networks:
  emission:

volumes:
  mongo-data:

However, when I try to build this docker container, I got

root@valued-byte-6:~/e-mission-docker/examples/em-server-multi-tier-cronjob# docker-compose -f docker-compose.yml up -d
WARNING: Some services (analysis-server, db, web-server) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Building analysis-server
Step 1/5 : FROM emission/e-mission-server.dev.server-only:2.9.1
 ---> 3220dce2de89
Step 2/5 : COPY conf /conf
ERROR: Service 'analysis-server' failed to build: COPY failed: file not found in build context or excluded by .dockerignore: stat conf: file does not exist

What should I do now?

I notice that seed_model.json is related to segmentation motion types. What happens if I ignore it?
Can I simply wget the seed_model_from_test_data.json (link) in a docker container built by the em-server/docker-compose.yml and run the analysis scripts?

shankari · 2021-02-23T04:17:19Z

@xubowenhaoren I would suggest using the GIS branch instead of master. Ad-hoc testing indicates that it works better in almost all cases. The docker containers support the branch for the server as an environment variable in docker-compose.
I am going to make the GIS branch the default at some point, but haven't got to it yet.

xubowenhaoren · 2021-02-23T07:59:28Z

Right, I tried to use the GIS branch in the mult-tier docker-compose file:

version: "3"
services:
  web-server:
    build: webapp
    depends_on:
      - db
    environment:
      - DB_HOST=db
      - WEB_SERVER_HOST=0.0.0.0
      - SERVER_BRANCH=gis-based-mode-detection
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    ports:
      # If you are exposing this directly, use 80:8080 or 443:8080
      # If you are using a reverse proxy, use a unique port and configure the
      # RP appropriately
      - "80:8080"
    networks:
      - emission
  analysis-server:
    build: analysis
    depends_on:
      - db
    environment:
      - DB_HOST=db
      - WEB_SERVER_HOST=0.0.0.0
      - SERVER_BRANCH=gis-based-mode-detection
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    networks:
      - emission
  db:
    image: mongo:4.4.0
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

    #Volumes is the preferred way to persist data generated by a container. In this case we use a volume to persist the contents
    #of the data base. Learn more about how to use volumes here: https://docs.docker.com/storage/volumes/
    # And learn how to configure volumes in your compose file here: https://docs.docker.com/compose/compose-file/#volume-configuration-reference
    volumes:
      - mongo-data:/data/db
    networks:
       - emission

networks:
  emission:

volumes:
  mongo-data:

However, when I try to build this docker container, I got

root@valued-byte-6:~/e-mission-docker/examples/em-server-multi-tier-cronjob# docker-compose -f docker-compose.yml up -d
WARNING: Some services (analysis-server, db, web-server) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
Building analysis-server
Step 1/5 : FROM emission/e-mission-server.dev.server-only:2.9.1
 ---> 3220dce2de89
Step 2/5 : COPY conf /conf
ERROR: Service 'analysis-server' failed to build: COPY failed: file not found in build context or excluded by .dockerignore: stat conf: file does not exist

I noticed that there's no conf file in the directory. I had to comment out this line to continue building the GIS-based docker.

xubowenhaoren · 2021-02-23T08:06:21Z

The docker emservermultitiercronjob_web-server died shortly after the deployment.

root@valued-byte-6:~/e-mission-docker/examples/em-server-multi-tier-cronjob# docker ps -a

CONTAINER ID   IMAGE                                      COMMAND                  CREATED         STATUS                     PORTS       NAMES
14fb95d3f6fc   emservermultitiercronjob_web-server        "/bin/bash /usr/src/…"   6 minutes ago   Exited (1) 4 minutes ago               emservermultitiercronjob_web-server_1
27b46fc95e09   emservermultitiercronjob_analysis-server   "/bin/bash /usr/src/…"   6 minutes ago   Up 6 minutes               8080/tcp    emservermultitiercronjob_analysis-server_1
d924a32a5d7e   mongo:4.4.0                                "docker-entrypoint.s…"   6 minutes ago   Up 6 minutes               27017/tcp   emservermultitiercronjob_db_1

The actual docker log:

root@valued-byte-6:~/e-mission-docker/examples/em-server-multi-tier-cronjob# docker logs 14fb95d3f6fc
Cloning from repo https://github.com/e-mission/e-mission-server.git and branch gis-based-mode-detection into /usr/src/app
Cloning into 'e-mission-server'...
Updating files: 100% (929/929), done.
Finished cloning from repo https://github.com/e-mission/e-mission-server.git and branch gis-based-mode-detection in /usr/src/app
From https://github.com/e-mission/e-mission-server
 * branch              gis-based-mode-detection -> FETCH_HEAD
Switched to a new branch 'gis-based-mode-detection'
Branch 'gis-based-mode-detection' set up to track remote branch 'gis-based-mode-detection' from 'origin'.
About to start conda update, this may take some time...
Installing for version 4.8.3 and platform Linux-x86_64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 88.7M  100 88.7M    0     0  61.3M      0  0:00:01  0:00:01 --:--:-- 68.2M
PREFIX=/root/miniconda-4.8.3
Unpacking payload ...
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /root/miniconda-4.8.3

  added / updated specs:
    - _libgcc_mutex==0.1=main
    - ca-certificates==2020.1.1=0
    - certifi==2020.4.5.1=py38_0
    - cffi==1.14.0=py38he30daa8_1
    - chardet==3.0.4=py38_1003
    - conda-package-handling==1.6.1=py38h7b6447c_0
    - conda==4.8.3=py38_0
    - cryptography==2.9.2=py38h1ba5d50_0
    - idna==2.9=py_1
    - ld_impl_linux-64==2.33.1=h53a641e_7
    - libedit==3.1.20181209=hc058e9b_0
    - libffi==3.3=he6710b0_1
    - libgcc-ng==9.1.0=hdf63c60_0
    - libstdcxx-ng==9.1.0=hdf63c60_0
    - ncurses==6.2=he6710b0_1
    - openssl==1.1.1g=h7b6447c_0
    - pip==20.0.2=py38_3
    - pycosat==0.6.3=py38h7b6447c_1
    - pycparser==2.20=py_0
    - pyopenssl==19.1.0=py38_0
    - pysocks==1.7.1=py38_0
    - python==3.8.3=hcff3b4d_0
    - readline==8.0=h7b6447c_0
    - requests==2.23.0=py38_0
    - ruamel_yaml==0.15.87=py38h7b6447c_0
    - setuptools==46.4.0=py38_0
    - six==1.14.0=py38_0
    - sqlite==3.31.1=h62c20be_1
    - tk==8.6.8=hbc83047_0
    - tqdm==4.46.0=py_0
    - urllib3==1.25.8=py38_0
    - wheel==0.34.2=py38_0
    - xz==5.2.5=h7b6447c_0
    - yaml==0.1.7=had09818_2
    - zlib==1.2.11=h7b6447c_3


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0
  certifi            pkgs/main/linux-64::certifi-2020.4.5.1-py38_0
  cffi               pkgs/main/linux-64::cffi-1.14.0-py38he30daa8_1
  chardet            pkgs/main/linux-64::chardet-3.0.4-py38_1003
  conda              pkgs/main/linux-64::conda-4.8.3-py38_0
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.1-py38h7b6447c_0
  cryptography       pkgs/main/linux-64::cryptography-2.9.2-py38h1ba5d50_0
  idna               pkgs/main/noarch::idna-2.9-py_1
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_1
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0
  pip                pkgs/main/linux-64::pip-20.0.2-py38_3
  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py38h7b6447c_1
  pycparser          pkgs/main/noarch::pycparser-2.20-py_0
  pyopenssl          pkgs/main/linux-64::pyopenssl-19.1.0-py38_0
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py38_0
  python             pkgs/main/linux-64::python-3.8.3-hcff3b4d_0
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  requests           pkgs/main/linux-64::requests-2.23.0-py38_0
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.87-py38h7b6447c_0
  setuptools         pkgs/main/linux-64::setuptools-46.4.0-py38_0
  six                pkgs/main/linux-64::six-1.14.0-py38_0
  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  tqdm               pkgs/main/noarch::tqdm-4.46.0-py_0
  urllib3            pkgs/main/linux-64::urllib3-1.25.8-py38_0
  wheel              pkgs/main/linux-64::wheel-0.34.2-py38_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  yaml               pkgs/main/linux-64::yaml-0.1.7-had09818_2
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: ...working... done
Executing transaction: ...working... done
installation finished.

     active environment : None
            shell level : 0
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.8.3
    conda-build version : not installed
         python version : 3.8.3.final.0
       virtual packages : __glibc=2.31
       base environment : /root/miniconda-4.8.3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /root/miniconda-4.8.3/pkgs
                          /root/.conda/pkgs
       envs directories : /root/miniconda-4.8.3/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.23.0 CPython/3.8.3 Linux/4.4.0-201-generic ubuntu/20.04 glibc/2.31
                UID:GID : 0:0
             netrc file : None
           offline mode : False

# conda environments:
#
base                  *  /root/miniconda-4.8.3

sys.version: 3.8.3 (default, May 19 2020, 18:47:26) 
...
sys.prefix: /root/miniconda-4.8.3
sys.executable: /root/miniconda-4.8.3/bin/python
conda location: /root/miniconda-4.8.3/lib/python3.8/site-packages/conda
conda-build: None
conda-env: /root/miniconda-4.8.3/bin/conda-env
user site dirs: 

CIO_TEST: <not set>
CONDA_EXE: /root/miniconda-4.8.3/bin/conda
CONDA_PYTHON_EXE: /root/miniconda-4.8.3/bin/python
CONDA_ROOT: /root/miniconda-4.8.3
CONDA_SHLVL: 0
PATH: /root/miniconda-4.8.3/bin:/root/miniconda-4.8.3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>

Successfully installed at /root/miniconda-4.8.3. Please activate with 'source setup/activateXXX.sh' in every terminal where you want to use conda
For conda, found 4.8.3, expected 4.8.3, all is good!
Installing using conda now
Collecting package metadata (repodata.json): ...working... failed
                                                                                                              
CondaMemoryError: The conda process ran out of memory. Increase system memory and/or try again.

Cache location: /root/miniconda-4.8.3/pkgs
Will remove the following tarballs:

/root/miniconda-4.8.3/pkgs
--------------------------
idna-2.9-py_1.conda                           49 KB
yaml-0.1.7-had09818_2.conda                   73 KB
xz-5.2.5-h7b6447c_0.conda                    341 KB
python-3.8.3-hcff3b4d_0.conda               49.1 MB
six-1.14.0-py38_0.conda                       27 KB
libstdcxx-ng-9.1.0-hdf63c60_0.conda          3.1 MB
certifi-2020.4.5.1-py38_0.conda              156 KB
ld_impl_linux-64-2.33.1-h53a641e_7.conda     568 KB
conda-package-handling-1.6.1-py38h7b6447c_0.conda     799 KB
conda-4.8.3-py38_0.tar.bz2                   3.0 MB
tqdm-4.46.0-py_0.conda                        60 KB
libffi-3.3-he6710b0_1.conda                   50 KB
zlib-1.2.11-h7b6447c_3.conda                 103 KB
tk-8.6.8-hbc83047_0.conda                    2.8 MB
libedit-3.1.20181209-hc058e9b_0.conda        163 KB
ncurses-6.2-he6710b0_1.conda                 817 KB
setuptools-46.4.0-py38_0.conda               515 KB
cffi-1.14.0-py38he30daa8_1.conda             225 KB
cryptography-2.9.2-py38h1ba5d50_0.conda      556 KB
libgcc-ng-9.1.0-hdf63c60_0.conda             5.1 MB
pycparser-2.20-py_0.conda                     92 KB
ca-certificates-2020.1.1-0.conda             125 KB
requests-2.23.0-py38_0.conda                  93 KB
_libgcc_mutex-0.1-main.conda                   3 KB
chardet-3.0.4-py38_1003.conda                174 KB
wheel-0.34.2-py38_0.conda                     51 KB
pysocks-1.7.1-py38_0.conda                    28 KB
sqlite-3.31.1-h62c20be_1.conda               1.1 MB
ruamel_yaml-0.15.87-py38h7b6447c_0.conda     259 KB
openssl-1.1.1g-h7b6447c_0.conda              2.5 MB
pyopenssl-19.1.0-py38_0.conda                 88 KB
readline-8.0-h7b6447c_0.conda                356 KB
pip-20.0.2-py38_3.conda                      1.7 MB
urllib3-1.25.8-py38_0.conda                  170 KB
pycosat-0.6.3-py38h7b6447c_1.conda            82 KB

---------------------------------------------------
Total:                                      74.3 MB

Removed idna-2.9-py_1.conda
Removed yaml-0.1.7-had09818_2.conda
Removed xz-5.2.5-h7b6447c_0.conda
Removed python-3.8.3-hcff3b4d_0.conda
Removed six-1.14.0-py38_0.conda
Removed libstdcxx-ng-9.1.0-hdf63c60_0.conda
Removed certifi-2020.4.5.1-py38_0.conda
Removed ld_impl_linux-64-2.33.1-h53a641e_7.conda
Removed conda-package-handling-1.6.1-py38h7b6447c_0.conda
Removed conda-4.8.3-py38_0.tar.bz2
Removed tqdm-4.46.0-py_0.conda
Removed libffi-3.3-he6710b0_1.conda
Removed zlib-1.2.11-h7b6447c_3.conda
Removed tk-8.6.8-hbc83047_0.conda
Removed libedit-3.1.20181209-hc058e9b_0.conda
Removed ncurses-6.2-he6710b0_1.conda
Removed setuptools-46.4.0-py38_0.conda
Removed cffi-1.14.0-py38he30daa8_1.conda
Removed cryptography-2.9.2-py38h1ba5d50_0.conda
Removed libgcc-ng-9.1.0-hdf63c60_0.conda
Removed pycparser-2.20-py_0.conda
Removed ca-certificates-2020.1.1-0.conda
Removed requests-2.23.0-py38_0.conda
Removed _libgcc_mutex-0.1-main.conda
Removed chardet-3.0.4-py38_1003.conda
Removed wheel-0.34.2-py38_0.conda
Removed pysocks-1.7.1-py38_0.conda
Removed sqlite-3.31.1-h62c20be_1.conda
Removed ruamel_yaml-0.15.87-py38h7b6447c_0.conda
Removed openssl-1.1.1g-h7b6447c_0.conda
Removed pyopenssl-19.1.0-py38_0.conda
Removed readline-8.0-h7b6447c_0.conda
Removed pip-20.0.2-py38_3.conda
Removed urllib3-1.25.8-py38_0.conda
Removed pycosat-0.6.3-py38h7b6447c_1.conda
WARNING: /root/.conda/pkgs does not exist
Cache location: /root/miniconda-4.8.3/pkgs
Will remove the following packages:
/root/miniconda-4.8.3/pkgs
--------------------------

_libgcc_mutex-0.1-main                         7 KB

---------------------------------------------------
Total:                                         7 KB

removing _libgcc_mutex-0.1-main
Found configuration, overriding...
Live reload disabled, 
db
{
    "timeseries": {
        "url": "db",
        "result_limit": 250000
    }
}
0.0.0.0
{
  "paths" : {
    "static_path" : "webapp/www/",
    "python_path" : "main",
    "log_base_dir" : ".",
    "log_file" : "debug.log"
  },
  "__comment" : "Fill this in for the production server. port will almost certainly be 80 or 443. For iOS, using 0.0.0.0 allows you to test without an internet connection. For AWS and android, make sure that the host 0.0.0.0, localhost does not seem to work",
  "server" : {
    "host" : "0.0.0.0",
    "port" : "8080",
    "__comment": "1 hour = 60 min = 60 * 60 sec",
    "timeout" : "3600",
    "auth": "skip"
  }
}
Live reload disabled, 
Could not find conda environment: emission
You can list all discoverable environments with `conda info --envs`.

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 6, in <module>
    from future import standard_library
ModuleNotFoundError: No module named 'future'
root@valued-byte-6:~/e-mission-docker/examples/em-server-multi-tier-cronjob# docker logs 14fb95d3f6fc
Cloning from repo https://github.com/e-mission/e-mission-server.git and branch gis-based-mode-detection into /usr/src/app
Cloning into 'e-mission-server'...
Updating files: 100% (929/929), done.
Finished cloning from repo https://github.com/e-mission/e-mission-server.git and branch gis-based-mode-detection in /usr/src/app
From https://github.com/e-mission/e-mission-server
 * branch              gis-based-mode-detection -> FETCH_HEAD
Switched to a new branch 'gis-based-mode-detection'
Branch 'gis-based-mode-detection' set up to track remote branch 'gis-based-mode-detection' from 'origin'.
About to start conda update, this may take some time...
Installing for version 4.8.3 and platform Linux-x86_64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 88.7M  100 88.7M    0     0  61.3M      0  0:00:01  0:00:01 --:--:-- 68.2M
PREFIX=/root/miniconda-4.8.3
Unpacking payload ...
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /root/miniconda-4.8.3

  added / updated specs:
    - _libgcc_mutex==0.1=main
    - ca-certificates==2020.1.1=0
    - certifi==2020.4.5.1=py38_0
    - cffi==1.14.0=py38he30daa8_1
    - chardet==3.0.4=py38_1003
    - conda-package-handling==1.6.1=py38h7b6447c_0
    - conda==4.8.3=py38_0
    - cryptography==2.9.2=py38h1ba5d50_0
    - idna==2.9=py_1
    - ld_impl_linux-64==2.33.1=h53a641e_7
    - libedit==3.1.20181209=hc058e9b_0
    - libffi==3.3=he6710b0_1
    - libgcc-ng==9.1.0=hdf63c60_0
    - libstdcxx-ng==9.1.0=hdf63c60_0
    - ncurses==6.2=he6710b0_1
    - openssl==1.1.1g=h7b6447c_0
    - pip==20.0.2=py38_3
    - pycosat==0.6.3=py38h7b6447c_1
    - pycparser==2.20=py_0
    - pyopenssl==19.1.0=py38_0
    - pysocks==1.7.1=py38_0
    - python==3.8.3=hcff3b4d_0
    - readline==8.0=h7b6447c_0
    - requests==2.23.0=py38_0
    - ruamel_yaml==0.15.87=py38h7b6447c_0
    - setuptools==46.4.0=py38_0
    - six==1.14.0=py38_0
    - sqlite==3.31.1=h62c20be_1
    - tk==8.6.8=hbc83047_0
    - tqdm==4.46.0=py_0
    - urllib3==1.25.8=py38_0
    - wheel==0.34.2=py38_0
    - xz==5.2.5=h7b6447c_0
    - yaml==0.1.7=had09818_2
    - zlib==1.2.11=h7b6447c_3


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0
  certifi            pkgs/main/linux-64::certifi-2020.4.5.1-py38_0
  cffi               pkgs/main/linux-64::cffi-1.14.0-py38he30daa8_1
  chardet            pkgs/main/linux-64::chardet-3.0.4-py38_1003
  conda              pkgs/main/linux-64::conda-4.8.3-py38_0
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.6.1-py38h7b6447c_0
  cryptography       pkgs/main/linux-64::cryptography-2.9.2-py38h1ba5d50_0
  idna               pkgs/main/noarch::idna-2.9-py_1
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_1
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1g-h7b6447c_0
  pip                pkgs/main/linux-64::pip-20.0.2-py38_3
  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py38h7b6447c_1
  pycparser          pkgs/main/noarch::pycparser-2.20-py_0
  pyopenssl          pkgs/main/linux-64::pyopenssl-19.1.0-py38_0
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py38_0
  python             pkgs/main/linux-64::python-3.8.3-hcff3b4d_0
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  requests           pkgs/main/linux-64::requests-2.23.0-py38_0
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.87-py38h7b6447c_0
  setuptools         pkgs/main/linux-64::setuptools-46.4.0-py38_0
  six                pkgs/main/linux-64::six-1.14.0-py38_0
  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h62c20be_1
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  tqdm               pkgs/main/noarch::tqdm-4.46.0-py_0
  urllib3            pkgs/main/linux-64::urllib3-1.25.8-py38_0
  wheel              pkgs/main/linux-64::wheel-0.34.2-py38_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  yaml               pkgs/main/linux-64::yaml-0.1.7-had09818_2
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: ...working... done
Executing transaction: ...working... done
installation finished.

     active environment : None
            shell level : 0
       user config file : /root/.condarc
 populated config files : /root/.condarc
          conda version : 4.8.3
    conda-build version : not installed
         python version : 3.8.3.final.0
       virtual packages : __glibc=2.31
       base environment : /root/miniconda-4.8.3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /root/miniconda-4.8.3/pkgs
                          /root/.conda/pkgs
       envs directories : /root/miniconda-4.8.3/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.23.0 CPython/3.8.3 Linux/4.4.0-201-generic ubuntu/20.04 glibc/2.31
                UID:GID : 0:0
             netrc file : None
           offline mode : False

# conda environments:
#
base                  *  /root/miniconda-4.8.3

sys.version: 3.8.3 (default, May 19 2020, 18:47:26) 
...
sys.prefix: /root/miniconda-4.8.3
sys.executable: /root/miniconda-4.8.3/bin/python
conda location: /root/miniconda-4.8.3/lib/python3.8/site-packages/conda
conda-build: None
conda-env: /root/miniconda-4.8.3/bin/conda-env
user site dirs: 

CIO_TEST: <not set>
CONDA_EXE: /root/miniconda-4.8.3/bin/conda
CONDA_PYTHON_EXE: /root/miniconda-4.8.3/bin/python
CONDA_ROOT: /root/miniconda-4.8.3
CONDA_SHLVL: 0
PATH: /root/miniconda-4.8.3/bin:/root/miniconda-4.8.3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>

Successfully installed at /root/miniconda-4.8.3. Please activate with 'source setup/activateXXX.sh' in every terminal where you want to use conda
For conda, found 4.8.3, expected 4.8.3, all is good!
Installing using conda now
Collecting package metadata (repodata.json): ...working... failed
                                                                                                              
CondaMemoryError: The conda process ran out of memory. Increase system memory and/or try again.

Cache location: /root/miniconda-4.8.3/pkgs
Will remove the following tarballs:

/root/miniconda-4.8.3/pkgs
--------------------------
idna-2.9-py_1.conda                           49 KB
yaml-0.1.7-had09818_2.conda                   73 KB
xz-5.2.5-h7b6447c_0.conda                    341 KB
python-3.8.3-hcff3b4d_0.conda               49.1 MB
six-1.14.0-py38_0.conda                       27 KB
libstdcxx-ng-9.1.0-hdf63c60_0.conda          3.1 MB
certifi-2020.4.5.1-py38_0.conda              156 KB
ld_impl_linux-64-2.33.1-h53a641e_7.conda     568 KB
conda-package-handling-1.6.1-py38h7b6447c_0.conda     799 KB
conda-4.8.3-py38_0.tar.bz2                   3.0 MB
tqdm-4.46.0-py_0.conda                        60 KB
libffi-3.3-he6710b0_1.conda                   50 KB
zlib-1.2.11-h7b6447c_3.conda                 103 KB
tk-8.6.8-hbc83047_0.conda                    2.8 MB
libedit-3.1.20181209-hc058e9b_0.conda        163 KB
ncurses-6.2-he6710b0_1.conda                 817 KB
setuptools-46.4.0-py38_0.conda               515 KB
cffi-1.14.0-py38he30daa8_1.conda             225 KB
cryptography-2.9.2-py38h1ba5d50_0.conda      556 KB
libgcc-ng-9.1.0-hdf63c60_0.conda             5.1 MB
pycparser-2.20-py_0.conda                     92 KB
ca-certificates-2020.1.1-0.conda             125 KB
requests-2.23.0-py38_0.conda                  93 KB
_libgcc_mutex-0.1-main.conda                   3 KB
chardet-3.0.4-py38_1003.conda                174 KB
wheel-0.34.2-py38_0.conda                     51 KB
pysocks-1.7.1-py38_0.conda                    28 KB
sqlite-3.31.1-h62c20be_1.conda               1.1 MB
ruamel_yaml-0.15.87-py38h7b6447c_0.conda     259 KB
openssl-1.1.1g-h7b6447c_0.conda              2.5 MB
pyopenssl-19.1.0-py38_0.conda                 88 KB
readline-8.0-h7b6447c_0.conda                356 KB
pip-20.0.2-py38_3.conda                      1.7 MB
urllib3-1.25.8-py38_0.conda                  170 KB
pycosat-0.6.3-py38h7b6447c_1.conda            82 KB

---------------------------------------------------
Total:                                      74.3 MB

Removed idna-2.9-py_1.conda
Removed yaml-0.1.7-had09818_2.conda
Removed xz-5.2.5-h7b6447c_0.conda
Removed python-3.8.3-hcff3b4d_0.conda
Removed six-1.14.0-py38_0.conda
Removed libstdcxx-ng-9.1.0-hdf63c60_0.conda
Removed certifi-2020.4.5.1-py38_0.conda
Removed ld_impl_linux-64-2.33.1-h53a641e_7.conda
Removed conda-package-handling-1.6.1-py38h7b6447c_0.conda
Removed conda-4.8.3-py38_0.tar.bz2
Removed tqdm-4.46.0-py_0.conda
Removed libffi-3.3-he6710b0_1.conda
Removed zlib-1.2.11-h7b6447c_3.conda
Removed tk-8.6.8-hbc83047_0.conda
Removed libedit-3.1.20181209-hc058e9b_0.conda
Removed ncurses-6.2-he6710b0_1.conda
Removed setuptools-46.4.0-py38_0.conda
Removed cffi-1.14.0-py38he30daa8_1.conda
Removed cryptography-2.9.2-py38h1ba5d50_0.conda
Removed libgcc-ng-9.1.0-hdf63c60_0.conda
Removed pycparser-2.20-py_0.conda
Removed ca-certificates-2020.1.1-0.conda
Removed requests-2.23.0-py38_0.conda
Removed _libgcc_mutex-0.1-main.conda
Removed chardet-3.0.4-py38_1003.conda
Removed wheel-0.34.2-py38_0.conda
Removed pysocks-1.7.1-py38_0.conda
Removed sqlite-3.31.1-h62c20be_1.conda
Removed ruamel_yaml-0.15.87-py38h7b6447c_0.conda
Removed openssl-1.1.1g-h7b6447c_0.conda
Removed pyopenssl-19.1.0-py38_0.conda
Removed readline-8.0-h7b6447c_0.conda
Removed pip-20.0.2-py38_3.conda
Removed urllib3-1.25.8-py38_0.conda
Removed pycosat-0.6.3-py38h7b6447c_1.conda
WARNING: /root/.conda/pkgs does not exist
Cache location: /root/miniconda-4.8.3/pkgs
Will remove the following packages:
/root/miniconda-4.8.3/pkgs
--------------------------

_libgcc_mutex-0.1-main                         7 KB

---------------------------------------------------
Total:                                         7 KB

removing _libgcc_mutex-0.1-main
Found configuration, overriding...
Live reload disabled, 
db
{
    "timeseries": {
        "url": "db",
        "result_limit": 250000
    }
}
0.0.0.0
{
  "paths" : {
    "static_path" : "webapp/www/",
    "python_path" : "main",
    "log_base_dir" : ".",
    "log_file" : "debug.log"
  },
  "__comment" : "Fill this in for the production server. port will almost certainly be 80 or 443. For iOS, using 0.0.0.0 allows you to test without an internet connection. For AWS and android, make sure that the host 0.0.0.0, localhost does not seem to work",
  "server" : {
    "host" : "0.0.0.0",
    "port" : "8080",
    "__comment": "1 hour = 60 min = 60 * 60 sec",
    "timeout" : "3600",
    "auth": "skip"
  }
}
Live reload disabled, 
Could not find conda environment: emission
You can list all discoverable environments with `conda info --envs`.

Traceback (most recent call last):
  File "emission/net/api/cfc_webapp.py", line 6, in <module>
    from future import standard_library
ModuleNotFoundError: No module named 'future'

Is this due to the missing conf or missing source setup/activate.sh?

shankari · 2021-02-23T18:30:00Z

Is this due to the missing conf or missing source setup/activate.sh?

Neither:

CondaMemoryError: The conda process ran out of memory. Increase system memory and/or try again.`

shankari · 2021-02-23T18:32:20Z

I noticed that there's no conf file in the directory. I had to comment out this line to continue building the GIS-based docker.

The instructions
https://github.com/e-mission/e-mission-docker/tree/master/examples/em-server-multi-tier-cronjob

say:

This is an example of a multi-tier setup, with the webapp, analysis and database tiers separated. The analysis tier is autoconfigured with a cronjob. There are multiple config options that need to be set before this container is run. These are:

you need to create conf directories in both webapp and analysis with the appropriate config files.

you should look through all the CHANGEME locations and fill them out

xubowenhaoren · 2021-02-24T05:56:06Z

Thank you for spotting the error. I deployed the GIS branch docker on a new instance with 2GB of RAM. Now I can run the analysis scripts with the expected exception of the missing habitica.json.

(emission) root@fae3a68cb77f:/usr/src/app/e-mission-server# ./e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py emission/tests/data/real_examples/shankari_2015-07-22 test_july_22
Connecting to database URL db
emission/tests/data/real_examples/shankari_2015-07-22
Loading file emission/tests/data/real_examples/shankari_2015-07-22
After registration, test_july_22 -> b1867891-d704-4e75-ada0-20098601bd61
Finished loading 0 entries into the usercache and 1906 entries into the timeseries
(emission) root@fae3a68cb77f:/usr/src/app/e-mission-server# ./e-mission-py.bash bin/debug/intake_single_user.py -e test_july_22
Connecting to database URL db
analysis.debug.conf.json not configured, falling back to sample, default configuration
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
overpass not configured, falling back to default overleaf.de
transit stops query not configured, falling back to default
ERROR:root:habitica not configured, game functions not supported
Traceback (most recent call last):
  File "/usr/src/app/e-mission-server/emission/net/ext_service/habitica/proxy.py", line 22, in <module>
    key_file = open('conf/net/ext_service/habitica.json')
FileNotFoundError: [Errno 2] No such file or directory: 'conf/net/ext_service/habitica.json'
2021-02-24T05:53:56.183796+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: moving to long term**********
2021-02-24T05:53:56.323602+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: updating incoming user inputs**********
2021-02-24T05:53:56.357925+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: filter accuracy if needed**********
2021-02-24T05:53:56.364802+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: segmenting into trips**********
2021-02-24T05:54:05.275679+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: segmenting into sections**********
2021-02-24T05:54:07.174328+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: smoothing sections**********
2021-02-24T05:54:08.126104+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: cleaning and resampling timeline**********
2021-02-24T05:54:10.727616+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: inferring transportation mode**********
2021-02-24T05:54:27.165420+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: creating confirmed objects **********
2021-02-24T05:54:27.239492+00:00**********UUID b1867891-d704-4e75-ada0-20098601bd61: storing views to cache**********

xubowenhaoren · 2021-02-24T06:24:28Z

I am having difficulties connecting my app to the new docker instance. I visited port 80 in the browser:

This is promising, but it's different than the e-mission website: https://e-mission.eecs.berkeley.edu/

When I set up the connectionConfig.json with the new URL and port combination, I get the error "HTTP 403 Forbidden":

shankari · 2021-02-24T14:59:09Z

When I set up the connectionConfig.json with the new URL and port combination, I get the error "HTTP 403 Forbidden":

You need to log out and log in again with the new server. Since you didn't register with it, your existing credentials don't authenticate with it.

wrt:

This is promising, but it's different than the e-mission website: https://e-mission.eecs.berkeley.edu/

I intentionally changed the boilerplate page to make things more modular:

not everybody is running emission.eecs.berkeley.edu
it adds a lot of overhead for the server (need to install bower, need to install javascript libraries)
it is against the principle of small micro-services.

xubowenhaoren · 2021-02-24T22:09:49Z

Thanks for the tip, I was able to sync my data to the server and get the trips segmented.

shankari changed the title ~~Cannot load the test data into the e-mission server docker~~ Fix bin/e-mission-py.bash in e-mission-docker to work with the new activation script Feb 22, 2021

xubowenhaoren closed this as completed Feb 24, 2021

MukuFlash03 mentioned this issue Mar 5, 2024

Unify buid and deploy processes across the various components of OpenPATH #1048

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `bin/e-mission-py.bash` in e-mission-docker to work with the new activation script #619

Fix `bin/e-mission-py.bash` in e-mission-docker to work with the new activation script #619

xubowenhaoren commented Feb 22, 2021

shankari commented Feb 22, 2021 •

edited

Loading

xubowenhaoren commented Feb 22, 2021 •

edited

Loading

shankari commented Feb 23, 2021

xubowenhaoren commented Feb 23, 2021

xubowenhaoren commented Feb 23, 2021

shankari commented Feb 23, 2021 •

edited

Loading

shankari commented Feb 23, 2021

xubowenhaoren commented Feb 24, 2021

xubowenhaoren commented Feb 24, 2021

shankari commented Feb 24, 2021

xubowenhaoren commented Feb 24, 2021

Fix bin/e-mission-py.bash in e-mission-docker to work with the new activation script #619

Fix bin/e-mission-py.bash in e-mission-docker to work with the new activation script #619

Comments

xubowenhaoren commented Feb 22, 2021

shankari commented Feb 22, 2021 • edited Loading

xubowenhaoren commented Feb 22, 2021 • edited Loading

shankari commented Feb 23, 2021

xubowenhaoren commented Feb 23, 2021

xubowenhaoren commented Feb 23, 2021

shankari commented Feb 23, 2021 • edited Loading

shankari commented Feb 23, 2021

xubowenhaoren commented Feb 24, 2021

xubowenhaoren commented Feb 24, 2021

shankari commented Feb 24, 2021

xubowenhaoren commented Feb 24, 2021

Fix `bin/e-mission-py.bash` in e-mission-docker to work with the new activation script #619

Fix `bin/e-mission-py.bash` in e-mission-docker to work with the new activation script #619

shankari commented Feb 22, 2021 •

edited

Loading

xubowenhaoren commented Feb 22, 2021 •

edited

Loading

shankari commented Feb 23, 2021 •

edited

Loading