Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On-board Aarya Patel #381

Closed
10 tasks done
samarth9008 opened this issue Jul 3, 2023 · 27 comments
Closed
10 tasks done

On-board Aarya Patel #381

samarth9008 opened this issue Jul 3, 2023 · 27 comments
Assignees

Comments

@samarth9008
Copy link
Collaborator

samarth9008 commented Jul 3, 2023

Please follow this checklist. Post any errors you face in this issue.

FYI @gpsaggese @DanilYachmenev

@aaryapatel100
Copy link

Just a question for quick start for developing, but is the project not able to be supported via WSL2 and WSL2-backed docker system in Windows?

@gpsaggese
Copy link
Contributor

Sorry, we only support Linux and Mac. For Windows we recommend installing Ubuntu in a VM. Running on Windows creates subtle bugs that are difficult to debug and not worth it.

@samarth9008
Copy link
Collaborator Author

Are you able to solve the docker problem?

@aaryapatel100
Copy link

Solved it for the most part with just installing VMWare and going the usual route but will let you know if I find a way to run a WSL-backed version of the container. Had some trouble trying to set things up as the terminal kept cache-locking due to unattended processes of security updates but fixed that and moving forward.

@gpsaggese
Copy link
Contributor

@aaryapatel100 do we need to improve our documentation of how to set up the Ubuntu set-up for windows users based on your experience?
@samarth9008 I think we had some doc, didn't we?

@aaryapatel100
Copy link

aaryapatel100 commented Jul 14, 2023

I think for the most part it's pretty good at pointing in the right direction, although clarification on some edge-cases of setting up (within the VM) could help new contributors getting set up faster. However, for the most part combination of directions and error searching is solid. Quick question - for docker desktop, I should install within the VM and not on my actual computer correct?

@samarth9008
Copy link
Collaborator Author

@aaryapatel100 do we need to improve our documentation of how to set up the Ubuntu set-up for windows users based on your experience? @samarth9008 I think we had some doc, didn't we?

It is mentioned in https://github.com/sorrentum/sorrentum/blob/master/docs/Quick_start_for_developing.md

@gpsaggese
Copy link
Contributor

I think for the most part it's pretty good at pointing in the right direction, although clarification on some edge-cases of setting up (within the VM) could help new contributors getting set up faster. However, for the most part combination of directions and error searching is solid.

Can you pls improve the doc, summarizing what you had to do and add links to what to download and how to install?

Quick question - for docker desktop, I should install within the VM and not on my actual computer correct?

Correct. It needs to be installed inside the VM. Pls also document the process here.

You can take screenshots, etc. We want to make sure everyone installs things in the same way.

@aaryapatel100
Copy link

I've built the thin environment and currently trying to start the jupyter server, but it keeps erroring out to 'srfs' is not defined. This is even after I ran the pip install s3fs command while within the virtual environment. Any suggestions on how to fix this?

@samarth9008
Copy link
Collaborator Author

@samarth9008
Copy link
Collaborator Author

@aaryapatel100
Copy link

  • Tried: (amp.client_venv) aaryapatel100@aaryapatel100VM:~/src/sorrentum1$ i docker_jupyter
  • Error output
    One and only one set-up config should be true: is_cmamp_prod=False is_dev4=False is_dev_ck=False is_ig_prod=False is_inside_ci=False is_mac=False WARNING: Can't find s3fs: continuing Traceback (most recent call last): File "/usr/bin/invoke", line 11, in <module> load_entry_point('invoke==1.4.1', 'console_scripts', 'invoke')() File "/usr/lib/python3/dist-packages/invoke/program.py", line 373, in run self.parse_collection() File "/usr/lib/python3/dist-packages/invoke/program.py", line 465, in parse_collection self.load_collection() File "/usr/lib/python3/dist-packages/invoke/program.py", line 696, in load_collection module, parent = loader.load(coll_name) File "/usr/lib/python3/dist-packages/invoke/loader.py", line 76, in load module = imp.load_module(name, fd, path, desc) File "/usr/lib/python3.10/imp.py", line 235, in load_module return load_source(name, filename, file) File "/usr/lib/python3.10/imp.py", line 172, in load_source module = _load(spec) File "<frozen importlib._bootstrap>", line 719, in _load File "<frozen importlib._bootstrap>", line 688, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/home/aaryapatel100/src/sorrentum1/tasks.py", line 10, in <module> from helpers.lib_tasks import ( # This is not an invoke target. File "/home/aaryapatel100/src/sorrentum1/helpers/lib_tasks.py", line 15, in <module> from helpers.lib_tasks_docker_release import * # isort:skip # noqa: F401,F403 # pylint: disable=unused-import,unused-wildcard-import,wildcard-import File "/home/aaryapatel100/src/sorrentum1/helpers/lib_tasks_docker_release.py", line 18, in <module> import helpers.hs3 as hs3 File "/home/aaryapatel100/src/sorrentum1/helpers/hs3.py", line 49, in <module> AwsProfile = Optional[Union[str, s3fs.core.S3FileSystem]] NameError: name 's3fs' is not defined
  • Problem: NameError: name 's3fs' is not defined even after running pip install s3fs within virutal environment.

@samarth9008
Copy link
Collaborator Author

How does the output look like while activating thin environment?

@aaryapatel100
Copy link

aaryapatel100@aaryapatel100VM:~/src/sorrentum1$ source dev_scripts/setenv_amp.sh
# Activate virtual env '/home/aaryapatel100/src/venv/amp.client_venv/bin/activate'
which python=/home/aaryapatel100/src/venv/amp.client_venv/bin/python
python -v=Python 3.10.6
# Set path
PATH=
/home/aaryapatel100/src/sorrentum1/dev_scripts/testing
/home/aaryapatel100/src/sorrentum1/dev_scripts/notebooks
/home/aaryapatel100/src/sorrentum1/dev_scripts/install
/home/aaryapatel100/src/sorrentum1/dev_scripts/infra
/home/aaryapatel100/src/sorrentum1/dev_scripts/git
/home/aaryapatel100/src/sorrentum1/dev_scripts/aws
/home/aaryapatel100/src/sorrentum1/dev_scripts
/home/aaryapatel100/src/sorrentum1
.
/home/aaryapatel100/src/venv/amp.client_venv/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games
/usr/local/games
/snap/bin
/snap/bin
# Set PYTHONPATH
PYTHONPATH=
/home/aaryapatel100/src/sorrentum1

# Configure env
which gh=
AM_AWS_PROFILE=am
AM_AWS_S3_BUCKET=alphamatic-data
AM_ECR_BASE_PATH=665840871993.dkr.ecr.us-east-1.amazonaws.com
CK_AWS_PROFILE=ck
CK_AWS_S3_BUCKET=cryptokaizen-data
alias alert='notify-send --urgency=low -i "$([ $? = 0 ] && echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&|]\s*alert$//'\'')"'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias grep='grep --color=auto'
alias i='invoke'
alias ih='invoke --help'
alias il='invoke --list'
alias it='invoke traceback'
alias itpb='pbpaste | traceback_to_cfile.py -i - -o cfile'
alias l='ls -CF'
alias la='ls -A'
alias ll='ls -alF'
alias ls='ls --color=auto'
==> SUCCESS <==

@samarth9008
Copy link
Collaborator Author

Can you format the i docker_jupyter output properly. Its hard to debug rn?

@samarth9008
Copy link
Collaborator Author

You can trying pulling a new images required for this.

@aaryapatel100
Copy link

Here's the formatted output:

(amp.client_venv) aaryapatel100@aaryapatel100VM:~/src/sorrentum1$ i docker_jupyter
One and only one set-up config should be true:
is_cmamp_prod=False
is_dev4=False
is_dev_ck=False
is_ig_prod=False
is_inside_ci=False
is_mac=False
WARNING: Can't find s3fs: continuing
Traceback (most recent call last):
  File "/usr/bin/invoke", line 11, in <module>
    load_entry_point('invoke==1.4.1', 'console_scripts', 'invoke')()
  File "/usr/lib/python3/dist-packages/invoke/program.py", line 373, in run
    self.parse_collection()
  File "/usr/lib/python3/dist-packages/invoke/program.py", line 465, in parse_collection
    self.load_collection()
  File "/usr/lib/python3/dist-packages/invoke/program.py", line 696, in load_collection
    module, parent = loader.load(coll_name)
  File "/usr/lib/python3/dist-packages/invoke/loader.py", line 76, in load
    module = imp.load_module(name, fd, path, desc)
  File "/usr/lib/python3.10/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.10/imp.py", line 172, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 719, in _load
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/aaryapatel100/src/sorrentum1/tasks.py", line 10, in <module>
    from helpers.lib_tasks import (  # This is not an invoke target.
  File "/home/aaryapatel100/src/sorrentum1/helpers/lib_tasks.py", line 15, in <module>
    from helpers.lib_tasks_docker_release import *  # isort:skip  # noqa: F401,F403 # pylint: disable=unused-import,unused-wildcard-import,wildcard-import
  File "/home/aaryapatel100/src/sorrentum1/helpers/lib_tasks_docker_release.py", line 18, in <module>
    import helpers.hs3 as hs3
  File "/home/aaryapatel100/src/sorrentum1/helpers/hs3.py", line 49, in <module>
    AwsProfile = Optional[Union[str, s3fs.core.S3FileSystem]]
NameError: name 's3fs' is not defined

@samarth9008
Copy link
Collaborator Author

I would suggest installing new images because this is something which is not inside your docker container if I am not wrong.

@aaryapatel100
Copy link

Quick update, but the issue stemmed from a build error that I fixed with an additional package added to the requirements.txt (details are within my documentation notes). But for the last step of quick_start_for_developing, am I supposed to run i docker_jupyter within the app?

Tried:
I have no name!@4c1233f3a49d:/app$ i docker_jupyter after running i docker_bash which resulted in bash: i: command not found.

Output for i docker_bash:

is_cmamp_prod=False
is_dev4=False
is_dev_ck=False
is_ig_prod=False
is_inside_ci=False
is_mac=False
INFO: > cmd='/home/aaryapatel100/src/venv/amp.client_venv/bin/invoke docker_bash'
## docker_bash: 
13:19:35 - WARN  <string> _raise_invalid_host:89                        Don't recognize host: host_os_name=Linux, am_host_os_name=None
13:19:35 - WARN  <string> _raise_invalid_host:89                        Don't recognize host: host_os_name=Linux, am_host_os_name=None
13:19:36 - WARN  <string> _raise_invalid_host:89                        Don't recognize host: host_os_name=Linux, am_host_os_name=None
13:19:36 - INFO  lib_tasks_docker.py _docker_cmd:1252                   Pulling the latest version of Docker
## docker_pull: 
## docker_login: 
  ... 
  ... The config profile (ck) could not be found
13:19:37 - INFO  lib_tasks_docker.py _docker_pull:226                   image='sorrentum/cmamp:dev'
docker pull sorrentum/cmamp:dev
dev: Pulling from sorrentum/cmamp
Digest: sha256:7d9ee52407e426c8d0c6611bebb3a5e76bf05d504122aaa50bf6765dc500a2f7
Status: Image is up to date for sorrentum/cmamp:dev
docker.io/sorrentum/cmamp:dev
IMAGE=sorrentum/cmamp:dev \
        docker-compose \
        --file /home/aaryapatel100/src/sorrentum1/devops/compose/docker-compose.yml \
        --env-file devops/env/default.env \
        run \
        --rm \
        --name aaryapatel100.cmamp.app.sorrentum1.20230721_111936 \
        --user $(id -u):$(id -g) \
        app \
        bash 
WARNING: The AM_AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The AM_AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The AM_AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The AM_FORCE_TEST_FAIL variable is not set. Defaulting to a blank string.
WARNING: The AM_TELEGRAM_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_ACCESS_KEY_ID variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_DEFAULT_REGION variable is not set. Defaulting to a blank string.
WARNING: The CK_AWS_SECRET_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The CK_TELEGRAM_TOKEN variable is not set. Defaulting to a blank string.
Creating network "compose_default" with the default driver
Creating compose_app_run ... done
##> devops/docker_run/entrypoint.sh
UID=1000
GID=138
# Activate environment
##> devops/docker_run/setenv.sh
# Set PATH
PATH=/app/documentation/scripts:/app/dev_scripts/testing:/app/dev_scripts/notebooks:/app/dev_scripts/install:/app/dev_scripts/infra:/app/dev_scripts/git:/app/dev_scripts/aws:/app/dev_scripts:/app:.:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Set PYTHONPATH
PYTHONPATH=/app:
# Configure env
git --version: git version 2.25.1
/app
WARNING: AWS credential check failed: can't find /home/.aws/credentials file.
WARNING: AWS credential check failed: can't find /home/.aws/config file.
# Check AWS authentication setup
      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                       am           manual    --profile

The config profile (am) could not be found
AM_CONTAINER_VERSION='1.4.0'
which python: /venv/bin/python
python -V: Python 3.8.10
helpers: <module 'helpers' from '/app/helpers/__init__.py'>
PATH=/app/documentation/scripts:/app/dev_scripts/testing:/app/dev_scripts/notebooks:/app/dev_scripts/install:/app/dev_scripts/infra:/app/dev_scripts/git:/app/dev_scripts/aws:/app/dev_scripts:/app:.:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH=/app:
entrypoint.sh: 'bash'
One and only one set-up config should be true:
is_cmamp_prod=False
is_dev4=False
is_dev_ck=False
is_ig_prod=False
is_inside_ci=False
is_mac=False
Don't recognize host: host_os_name=Linux, am_host_os_name=Linux
INFO: > cmd='/venv/bin/invoke print_env'
13:19:57 - WARN  <string> _raise_invalid_host:89                        Don't recognize host: host_os_name=Linux, am_host_os_name=Linux
13:19:57 - WARN  <string> _raise_invalid_host:89                        Don't recognize host: host_os_name=Linux, am_host_os_name=Linux
13:19:57 - WARN  <string> _raise_invalid_host:89                        Don't recognize host: host_os_name=Linux, am_host_os_name=Linux
13:19:57 - WARN  <string> _raise_invalid_host:89                        Don't recognize host: host_os_name=Linux, am_host_os_name=Linux
13:19:57 - WARN  <string> _raise_invalid_host:89                        Don't recognize host: host_os_name=Linux, am_host_os_name=Linux
-----------------------------------------------------------------------------
This code is not in sync with the container:
code_version='1.4.3' != container_version='1.4.0'
-----------------------------------------------------------------------------
You need to:
- merge origin/master into your branch with `invoke git_merge_master`
- pull the latest container with `invoke docker_pull`
# Repo config:
  # repo_config.config
    enable_privileged_mode='False'
    get_docker_base_image_name='cmamp'
    get_docker_shared_group=''
    get_docker_user=''
    get_host_name='github.com'
    get_html_dir_to_url_mapping='{'s3://cryptokaizen-html': 'http://172.30.2.44'}'
    get_invalid_words='[]'
    get_name='//sorr'
    get_repo_map='{'sorr': 'sorrentum/sorrentum'}'
    get_shared_data_dirs='None'
    has_dind_support='False'
    has_docker_sudo='False'
    is_CK_S3_available='True'
    run_docker_as_root='False'
    skip_submodules_test='False'
    use_docker_db_container_name_to_connect='False'
    use_docker_network_mode_host='False'
    use_docker_sibling_containers='False'
    # hserver.config
      is_AM_S3_available()='True'
      is_dev4()='False'
      is_dev_ck()='False'
      is_inside_ci()='False'
      is_inside_docker()='True'
      is_mac(version='Catalina')='False'
      is_mac(version='Monterey')='False'
      is_mac(version='Ventura')='False'
# System signature:
  # Git
    branch_name='master'
    hash='5b4261208'
    # Last commits:
      * 5b4261208 Samarth KaPatel Update Signing_up_for_Sorrentum.md (#436)                         (69 minutes ago) Fri Jul 21 16:11:25 2023  (HEAD -> master, origin/master, origin/HEAD)
      * 624287884 dchoi127 PR for SorrTask395_Unit_test_plotting_gallery_notebook (#402)     (    3 days ago) Tue Jul 18 18:18:11 2023           
      * 6fcda3bf6 hhxjqm   SorrTask408_Unit_test_plot_time_series_by_period (#414)           (    4 days ago) Tue Jul 18 00:47:02 2023           
  # Machine info
    system=Linux
    node name=4c1233f3a49d
    release=5.19.0-46-generic
    version=#47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2
    machine=x86_64
    processor=x86_64
    cpu count=4
    cpu freq=scpufreq(current=2918.4, min=0.0, max=0.0)
    memory=svmem(total=8322441216, available=3143442432, percent=62.2, used=4685336576, free=139071488, active=1551376384, inactive=4158083072, buffers=304300032, cached=3193733120, shared=184123392, slab=716083200)
    disk usage=sdiskusage(total=104556617728, used=23123275776, free=76074909696, percent=23.3)
  # Packages
    python: 3.8.10
    cvxopt: 1.3.0
    cvxpy: 1.2.2
    gluonnlp: ?
    gluonts: 0.6.7
    joblib: 1.2.0
    mxnet: 1.9.1
    numpy: 1.23.4
    pandas: 1.5.1
    pyarrow: 10.0.0
    scipy: 1.9.3
    seaborn: 0.12.1
    sklearn: 1.1.3
    statsmodels: 0.13.5
# Env vars:
  AM_AWS_ACCESS_KEY_ID=undef
  AM_AWS_DEFAULT_REGION=undef
  AM_AWS_PROFILE='am'
  AM_AWS_S3_BUCKET='alphamatic-data'
  AM_AWS_SECRET_ACCESS_KEY=undef
  AM_ECR_BASE_PATH='665840871993.dkr.ecr.us-east-1.amazonaws.com'
  AM_ENABLE_DIND='0'
  AM_FORCE_TEST_FAIL=''
  AM_HOST_NAME='aaryapatel100VM'
  AM_HOST_OS_NAME='Linux'
  AM_HOST_USER_NAME='aaryapatel100'
  AM_HOST_VERSION='5.19.0-46-generic'
  AM_REPO_CONFIG_CHECK='True'
  AM_REPO_CONFIG_PATH=''
  AM_TELEGRAM_TOKEN=empty
  CI=''
  CK_AWS_ACCESS_KEY_ID=empty
  CK_AWS_DEFAULT_REGION=''
  CK_AWS_S3_BUCKET='cryptokaizen-data'
  CK_AWS_SECRET_ACCESS_KEY=empty
  CK_ECR_BASE_PATH='sorrentum'
  GH_ACTION_ACCESS_TOKEN=empty

13:19:58 - INFO  hcache.py clear_global_cache:292                       Before clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=32.0 KB
13:19:58 - WARN  hcache.py clear_global_cache:293                       Resetting 'global mem' cache '/mnt/tmpfs/tmp.cache.mem'
13:19:58 - WARN  hcache.py clear_global_cache:303                       Destroying '/mnt/tmpfs/tmp.cache.mem' ...
13:19:58 - INFO  hcache.py clear_global_cache:319                       After clear_global_cache: 'global mem' cache: path='/mnt/tmpfs/tmp.cache.mem', size=nan
groups: cannot find name for group ID 138

@samarth9008
Copy link
Collaborator Author

No within the local environment.

@aaryapatel100
Copy link

Ok, finished the quick developing task. Here are all the steps that I added for setting up the VM (wasn't sure if you wanted me to directly update documentation):

  • Install GIT; If any instances of cache-locking occur in terminal follow steps within https://itsfoss.com/could-not-get-lock-error/#troubleshooting-1-unable-to-acquire-the-dpkg-frontend-lock-. If an unattended process is taking place, often being a security patch, restart VM to install patch updates. Then retry git installation process.

  • Install linux version of docker within VM (with given documentation link)

  • If on Ubuntu VM, run mkdir src within home directory and clone repository within src directory.

  • Prior to running/building thin environment, ensure that python3.10-venv is installed
    (very important). If not installed, run sudo apt install python3.10-venv

  • Run the build command: source dev_scripts/client_setup/build.sh

  • If encountering permission denied within venv:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied

follow steps on https://phoenixnap.com/kb/docker-permission-denied.

After running i docker_bash, run exit out of app terminal (I have no name!@4c1233f3a49d:/app$ exit).

Follow documentation steps for starting jupyter server and ensure everything works.

@aaryapatel100
Copy link

Also, I managed to through all the quick-developing steps using the WSL2-backed setup as well on Windows. Was just wondering if the bugs with running this project on Windows were later down the line or should be good to work if the quick developing functions?

@gpsaggese
Copy link
Contributor

  1. Good job. Yes pls update the documentation directly with a PR (try to follow the style / convention we use)

  2. In brief, we have found some intermittent issues with Docker WSL, both with Sorrentum and with UMD Data605. Overall we could support WSL but the issue is: is it worth to risk hitting weird errors when one can just use Linux? Also Windows is rarely used, if ever, in research. If a company uses Windows, I would not even apply there. Even people at Micro$oft use Macs. Windows is responsible for a rent-seeking monopoly that transferred a huge amount of capital from the society at large to Bill Gate$, that now goes around acting as a world benefactor. Windows is just the proof that an inferior product, together with predatory techniques can succeed in a distorted market place. All this considered, no we don't support Windows.

@samarth9008
Copy link
Collaborator Author

If you are done with everything mark as done and feel free to close this issue.

@aaryapatel100
Copy link

Ok, marked everything besides the warm-up issue which I haven't been assigned yet.

@samarth9008
Copy link
Collaborator Author

Assigned.
#491

@samarth9008
Copy link
Collaborator Author

Closing as completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants