TutorTask394_DataLoader_for_Historical_Causal_AutoML_and_FRED by indrayudd · Pull Request #404 · causify-ai/tutorials

indrayudd · 2025-05-07T08:28:13Z

This is PR1 for #394 where I wrap a class FredDataLoader around a third-party api called fredapi.

It has a function load_series with a contract as sought. I didn't do an interface push for this one because @sonniki already mentioned the kind of interface that was sought with the class.

I was a little doubtful about where to locate the file, but for the time being I put it in the root of causal_automl itself, where eda_utils exists. To this, I would add caching for PR2.

…git discrepancy. Please ignore

…sal_AutoML_and_FRED

sonniki · 2025-05-09T22:17:32Z

@@ -0,0 +1,130 @@
+import logging


BTW - we are missing the baptizing statement ("Import as..."). Doesn't Linter add it automatically?

No, it didn't. Manually making the change though.

There is still no baptizing statement... Pls refer to https://github.com/causify-ai/helpers/blob/c7a9c5777a8f337f43e84d6fc7fc511e48153207/docs/code_guidelines/all.coding_style.how_to_guide.md#baptizing-module-import

Are you sure Linter doesn't generate it? Pls try again with Linter. It generates it for me in files in this repo, just as it should.

My Linter does show me the warnings, but no it didn't generate the statements. Am I invoking it wrong? I'm using i lint --files=...

Here's the action output for your reference:

//////////////////////////////////////////////////////////////////////////////// ./linter_warnings.txt //////////////////////////////////////////////////////////////////////////////// cmd line='./helpers_root/linters/base.py --files causal_automl/download_fred_data.py --num_threads serial' file_paths=1 ['causal_automl/download_fred_data.py'] actions=25 ['add_python_init_files', 'add_toc_to_notebook', 'fix_md_links', 'lint_md', 'check_md_toc_headers', 'autoflake', 'fix_whitespaces', 'doc_formatter', 'isort', 'class_method_order', 'normalize_imports', 'format_separating_line', 'add_class_frames', 'remove_empty_lines_in_function', 'black', 'process_jupytext', 'check_file_size', 'check_filename', 'check_merge_conflict', 'check_import', 'warn_incorrectly_formatted_todo', 'check_md_reference', 'flake8', 'pylint', 'mypy']

Weird... I assume, the master is updated and merged into your branch, and the thin env is activated. Could you please provide the full stacktrace, so, including your command and until the very end of the terminal basically (BTW you should always include your command when reporting errors and such, so that people can reproduce).

Just to double check, when you run Linter on this file now, it doesn't stage it for commit? Or does it?

@sonniki I deleted my old src files, freshly pulled tutorials and reran i docker_pull and i docker_pull_helpers. Linter still failed to add the baptising statement. Is there anything more I could purge before trying a truly fresh install of the environment? My code editor btw is VS Code. Here's the freshest trace after doing all this:

(client_venv.helpers) (base) Indrayudds-MacBook-Air:tutorials1 indro$ i lint --files=causal_automl/download_fred_data.py 04:17:26 - INFO hdbg.py init_logger:1018 > cmd='/Users/indro/src/venv/client_venv.helpers/bin/invoke lint --files=causal_automl/download_fred_data.py' # lint: base_image='', stage='prod', version='', files='causal_automl/download_fred_data.py', skip_files='', dir_name='', modified=False, last_commit=False, branch=False, num_threads='-1', only_format=False, only_check=False 04:17:26 - INFO lib_tasks_lint.py lint:274 All Linter actions selected IMAGE=causify/helpers:prod \ docker compose \ --file /Users/indro/src/tutorials1/devops/compose/tmp.docker-compose.yml \ --env-file devops/env/default.env \ run \ --rm \ --name indro.tutorials.app.tutorials1.20250520_001726 \ --user $(id -u):$(id -g) \ app \ $(find . -path '*linters/base.py') --files causal_automl/download_fred_data.py --num_threads -1 time="2025-05-20T00:17:26-04:00" level=warning msg="The \"CSFY_FORCE_TEST_FAIL\" variable is not set. Defaulting to a blank string." time="2025-05-20T00:17:26-04:00" level=warning msg="The \"CSFY_AWS_ACCESS_KEY_ID\" variable is not set. Defaulting to a blank string." time="2025-05-20T00:17:26-04:00" level=warning msg="The \"CSFY_AWS_DEFAULT_REGION\" variable is not set. Defaulting to a blank string." time="2025-05-20T00:17:26-04:00" level=warning msg="The \"CSFY_AWS_SECRET_ACCESS_KEY\" variable is not set. Defaulting to a blank string." time="2025-05-20T00:17:26-04:00" level=warning msg="The \"CSFY_AWS_SESSION_TOKEN\" variable is not set. Defaulting to a blank string." time="2025-05-20T00:17:26-04:00" level=warning msg="The \"CSFY_TELEGRAM_TOKEN\" variable is not set. Defaulting to a blank string." time="2025-05-20T00:17:26-04:00" level=warning msg="/Users/indro/src/tutorials1/devops/compose/tmp.docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion" ##> devops/docker_run/entrypoint.sh UID=501 GID=20 CSFY_USE_HELPERS_AS_NESTED_MODULE=1 CSFY_HOST_GIT_ROOT_PATH=/Users/indro/src/tutorials1 CSFY_GIT_ROOT_PATH=/app CSFY_HELPERS_ROOT_PATH=/app/helpers_root > source /app/helpers_root/dev_scripts_helpers/thin_client/thin_client_utils.sh ... AM_CONTAINER_VERSION='1.3.0' CSFY_USE_HELPERS_AS_NESTED_MODULE=1 ##> devops/docker_run/docker_setenv.sh > source /app/helpers_root/dev_scripts_helpers/thin_client/thin_client_utils.sh ... # activate_docker_venv() # set_path() PATH=.:./dev_scripts_tutorials:./tutorial_openai:./devops:./tutorial_pymc:./tutorial_github_causify_style:./tutorial_langchain:./tutorial_neo4j:./sorrentum:./docker_common:./docs:./helpers_root:./causal_automl:./.mypy_cache:./UMD_MSML610:./.github:./tutorial_prophet:./tutorial_data_science:./llms:./.git:./tutorial_asana:./tutorial_github_data605_style:./DATA605:./tutorial_tensorflow::/app:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # set_up_docker_git() git --version: git version 2.43.0 /app # set_pythonpath() Adding /app/helpers_root to PYTHONPATH Adding /app to PYTHONPATH PYTHONPATH=/app:/app/helpers_root: # Configure env Setting sudo docker permissions srw-rw---- 1 root root 0 May 20 04:13 /var/run/docker.sock srw-rw-rw- 1 root root 0 May 20 04:13 /var/run/docker.sock # set_up_docker_git() git --version: git version 2.43.0 /app # invoke print_env 00:17:30 - INFO hdbg.py init_logger:1018 > cmd='/venv/bin/invoke print_env' 00:17:32 - INFO font_manager.py _load_fontmanager:1639 generated new fontManager # Repo config get_host_name='github.com' get_html_dir_to_url_mapping='{'s3://cryptokaizen-html': 'http://172.30.2.44', 's3://cryptokaizen-html/v2': 'http://172.30.2.44/v2'}' get_invalid_words='[]' get_docker_base_image_name='tutorials' # Server config enable_privileged_mode='False' get_docker_shared_group='' get_docker_user='' get_host_user_name='indro' get_shared_data_dirs='None' has_dind_support='False' has_docker_sudo='True' is_AM_S3_available='True' is_CK_S3_available='True' is_dev4='False' is_dev_csfy='False' is_external_linux='False' is_host_mac='True' is_ig_prod='False' is_inside_ci='False' is_inside_docker='True' is_inside_ecs_container='False' is_inside_unit_test='False' is_prod_csfy='False' run_docker_as_root='False' skip_submodules_test='False' use_docker_db_container_name_to_connect='True' use_docker_network_mode_host='False' use_docker_sibling_containers='True' use_main_network='True' # System signature # Container version container_version='1.3.0' changelog_version='1.0.0' # Git info branch_name='master' hash='b9e6840cd' # Last commits: * b9e6840cd Chanakya TutorTask110_Spring2025_Real-Time_Bitcoin_Analysis_Using_Transformers (#148) ( 3 hours ago) Tue May 20 01:16:38 2025 (HEAD -> master, origin/master, origin/HEAD) * 2871123fc amysamy1611 TutorTask226_Spring2025_Altair_Data_Visualization (#321) ( 5 hours ago) Mon May 19 23:05:15 2025 * e53f4c3d4 Indro TutorTask394_DataLoader_for_Historical_Causal_AutoML_and_FRED (#404) ( 7 days ago) Tue May 13 13:20:08 2025 (origin/fix_after_pull, origin/Spring2025_Real_Time_Bitcoin_Data_Analysis_with_Presto_461, origin/149, origin/147, origin/105) # Platform info system=Linux node name=bdf67aeec7e5 release=6.10.14-linuxkit version=#1 SMP Mon Feb 24 16:35:16 UTC 2025 machine=aarch64 processor=aarch64 # psutils info cpu count=8 cpu freq=None memory=svmem(total=4109803520, available=3477676032, percent=15.4, used=487112704, free=3075866624, active=686964736, inactive=159870976, buffers=50364416, cached=496459776, shared=630784, slab=53981184) disk usage=sdiskusage(total=1081100128256, used=17379184640, free=1008728637440, percent=1.7) # Docker info has_docker=True docker_version='28.0.1' docker_needs_sudo=False has_privileged_mode=True is_inside_docker=True has_sibling_containers_support=True has_docker_dind_support=True # Packages python: 3.12.3 cvxopt: ? cvxpy: ? gluonnlp: ? gluonts: ? joblib: 1.5.0 mxnet: ? numpy: 2.2.5 pandas: 2.2.3 pyarrow: 20.0.0 scipy: 1.15.3 seaborn: 0.13.2 sklearn: 1.6.1 statsmodels: 0.14.4 # Env vars CSFY_AWS_ACCESS_KEY_ID=empty CSFY_AWS_DEFAULT_REGION='' CSFY_AWS_S3_BUCKET='cryptokaizen-data' CSFY_AWS_SECRET_ACCESS_KEY=empty CSFY_AWS_SESSION_TOKEN='' CSFY_CI='' CSFY_ECR_BASE_PATH='causify' CSFY_ENABLE_DIND='0' CSFY_FORCE_TEST_FAIL='' CSFY_HOST_NAME='Indrayudds-MacBook-Air.local' CSFY_HOST_OS_NAME='Darwin' CSFY_HOST_USER_NAME='indro' CSFY_HOST_VERSION='24.3.0' CSFY_REPO_CONFIG_CHECK='True' CSFY_REPO_CONFIG_PATH='' GH_ACTION_ACCESS_TOKEN=empty PATH=.:./dev_scripts_tutorials:./tutorial_openai:./devops:./tutorial_pymc:./tutorial_github_causify_style:./tutorial_langchain:./tutorial_neo4j:./sorrentum:./docker_common:./docs:./helpers_root:./causal_automl:./.mypy_cache:./UMD_MSML610:./.github:./tutorial_prophet:./tutorial_data_science:./llms:./.git:./tutorial_asana:./tutorial_github_data605_style:./DATA605:./tutorial_tensorflow::/app:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PYTHONPATH=/app:/app/helpers_root: # entrypoint.sh: './helpers_root/linters/base.py --files causal_automl/download_fred_data.py --num_threads -1' 00:17:38 - INFO hdbg.py init_logger:1018 > cmd='./helpers_root/linters/base.py --files causal_automl/download_fred_data.py --num_threads -1' 00:17:41 - INFO base.py _run_linter:401 Using num_threads='serial' since there is only one file to lint 00:17:41 - INFO base.py _lint:363 Linting file: 'causal_automl/download_fred_data.py' //////////////////////////////////////////////////////////////////////////////// ./linter_warnings.txt //////////////////////////////////////////////////////////////////////////////// cmd line='./helpers_root/linters/base.py --files causal_automl/download_fred_data.py --num_threads -1' file_paths=1 ['causal_automl/download_fred_data.py'] actions=25 ['add_python_init_files', 'add_toc_to_notebook', 'fix_md_links', 'lint_md', 'check_md_toc_headers', 'autoflake', 'fix_whitespaces', 'doc_formatter', 'isort', 'class_method_order', 'normalize_imports', 'format_separating_line', 'add_class_frames', 'remove_empty_lines_in_function', 'black', 'process_jupytext', 'check_file_size', 'check_filename', 'check_merge_conflict', 'check_import', 'warn_incorrectly_formatted_todo', 'check_md_reference', 'flake8', 'pylint', 'mypy'] //////////////////////////////////////////////////////////////////////////////// causal_automl/download_fred_data.py:6: error: Cannot find implementation or library stub for module named "fredapi" [import-not-found] [mypy] causal_automl/download_fred_data.py:99: [W0718(broad-exception-caught), FredDataDownloader.download_series] Catching too general exception Exception [pylint] causal_automl/download_fred_data.py:9: error: Cannot find implementation or library stub for module named "ratelimit" [import-not-found] [mypy] ////////////////////////////////////////////////////////////////////////////////

The commit history doesn't look like it's outdated anymore; but that doesn't fix the issue. Is there something I'm missing? Is there a way to completely purge and reinstall the docker image side of things? Maybe there's a mismatch there? I'm not sure how exactly to proceed with that, because my original impression was that deleting src will remove everything.

I'm afraid I cannot propose a fix from my side, it's pretty impossible to debug setup issues remotely... Below is my stacktrace running Linter on the server, after pulling latest Docker image and master (it does add the baptizing statement for me). Maybe you'll notice some discrepancies with yours that will give you an idea about how to debug. Purging everything and going through the environment setup process from scratch is also an option.

(client_venv.helpers) sonyan@dev2:~/src/tutorials1$ i lint --files="causal_automl/download_fred_data.py" 08:22:55 - INFO hdbg.py init_logger:1018 > cmd='/data/sonyan/src/venv/client_venv.helpers/bin/invoke lint --files=causal_automl/download_fred_data.py' # lint: base_image='', stage='prod', version='', files='causal_automl/download_fred_data.py', skip_files='', dir_name='', modified=False, last_commit=False, branch=False, num_threads='serial', only_format=False, only_check=False 08:22:55 - INFO lib_tasks_lint.py lint:274 All Linter actions selected IMAGE=623860924167.dkr.ecr.eu-north-1.amazonaws.com/helpers:prod \ docker compose \ --file /data/sonyan/src/tutorials1/devops/compose/tmp.docker-compose.yml \ --env-file devops/env/default.env \ run \ --rm \ --name sonyan.tutorials.app.tutorials1.20250520_082255 \ --user $(id -u):$(id -g) \ app \ $(find -wholename '*linters/base.py') --files causal_automl/download_fred_data.py --num_threads serial time="2025-05-20T08:22:55Z" level=warning msg="The \"CSFY_FORCE_TEST_FAIL\" variable is not set. Defaulting to a blank string." time="2025-05-20T08:22:55Z" level=warning msg="The \"CSFY_AWS_ACCESS_KEY_ID\" variable is not set. Defaulting to a blank string." time="2025-05-20T08:22:55Z" level=warning msg="The \"CSFY_AWS_DEFAULT_REGION\" variable is not set. Defaulting to a blank string." time="2025-05-20T08:22:55Z" level=warning msg="The \"CSFY_AWS_SECRET_ACCESS_KEY\" variable is not set. Defaulting to a blank string." time="2025-05-20T08:22:55Z" level=warning msg="The \"CSFY_AWS_SESSION_TOKEN\" variable is not set. Defaulting to a blank string." time="2025-05-20T08:22:55Z" level=warning msg="The \"CSFY_TELEGRAM_TOKEN\" variable is not set. Defaulting to a blank string." time="2025-05-20T08:22:55Z" level=warning msg="The \"OPENAI_API_KEY\" variable is not set. Defaulting to a blank string." time="2025-05-20T08:22:55Z" level=warning msg="/data/sonyan/src/tutorials1/devops/compose/tmp.docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion" ##> devops/docker_run/entrypoint.sh UID=1013 GID=998 CSFY_USE_HELPERS_AS_NESTED_MODULE=1 CSFY_HOST_GIT_ROOT_PATH=/data/sonyan/src/tutorials1 CSFY_GIT_ROOT_PATH=/app CSFY_HELPERS_ROOT_PATH=/app/helpers_root > source /app/helpers_root/dev_scripts_helpers/thin_client/thin_client_utils.sh ... AM_CONTAINER_VERSION='1.3.0' CSFY_USE_HELPERS_AS_NESTED_MODULE=1 ##> devops/docker_run/docker_setenv.sh > source /app/helpers_root/dev_scripts_helpers/thin_client/thin_client_utils.sh ... # activate_docker_venv() # set_path() PATH=.:./.git:./helpers_root:./docker_common:./causal_automl:./tutorial_tensorflow:./tutorial_pymc:./llms:./dev_scripts_tutorials:./UMD_MSML610:./tutorial_github_data605_style:./tutorial_langchain:./.mypy_cache:./tutorial_openai:./docs:./tutorial_github_causify_style:./tutorial_prophet:./devops:./sorrentum:./tutorial_data_science:./DATA605:./tutorial_neo4j:./tutorial_asana:./.github::/app:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # set_up_docker_git() git --version: git version 2.43.0 /app # set_pythonpath() Adding /app/helpers_root to PYTHONPATH Adding /app to PYTHONPATH PYTHONPATH=/app:/app/helpers_root: # Configure env # set_up_docker_in_docker() { "storage-driver": "vfs" } * Starting Docker: docker ...done. * Docker is running Waiting for /var/run/docker.sock to be created. Permissions for /var/run/docker.sock have been changed. Setting sudo docker permissions srw-rw-rw- 1 root docker 0 May 20 08:22 /var/run/docker.sock srw-rw-rw- 1 root docker 0 May 20 08:22 /var/run/docker.sock # set_up_docker_git() git --version: git version 2.43.0 /app # invoke print_env 04:22:59 - INFO hdbg.py init_logger:1018 > cmd='/venv/bin/invoke print_env' Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world e6590344b1a5: Pulling fs layer e6590344b1a5: Verifying Checksum e6590344b1a5: Download complete e6590344b1a5: Pull complete Digest: sha256:dd01f97f252193ae3210da231b1dca0cffab4aadb3566692d6730bf93f123a48 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/ 04:23:03 - INFO font_manager.py _load_fontmanager:1639 generated new fontManager # Repo config get_host_name='github.com' get_html_dir_to_url_mapping='{'s3://cryptokaizen-html': 'http://172.30.2.44', 's3://cryptokaizen-html/v2': 'http://172.30.2.44/v2'}' get_invalid_words='[]' get_docker_base_image_name='tutorials' # Server config enable_privileged_mode='True' get_docker_shared_group='' get_docker_user='' get_host_user_name='sonyan' get_shared_data_dirs='{'/data/shared': '/shared_data', '/data/shared2': '/shared_data2', '/data/shared_k8s': '/shared_k8s'}' has_dind_support='True' has_docker_sudo='True' is_AM_S3_available='True' is_CK_S3_available='True' is_dev4='False' is_dev_csfy='True' is_external_linux='False' is_host_mac='False' is_ig_prod='False' is_inside_ci='False' is_inside_docker='True' is_inside_ecs_container='False' is_inside_unit_test='False' is_prod_csfy='False' run_docker_as_root='False' skip_submodules_test='False' use_docker_db_container_name_to_connect='False' use_docker_network_mode_host='False' use_docker_sibling_containers='False' use_main_network='False' # System signature # Container version container_version='1.3.0' changelog_version='1.0.0' # Git info branch_name='master' hash='b9e6840cd' # Last commits: * b9e6840cd Chanakya TutorTask110_Spring2025_Real-Time_Bitcoin_Analysis_Using_Transformers (#148) ( 7 hours ago) Tue May 20 01:16:38 2025 (HEAD -> master, origin/master, origin/HEAD) * 2871123fc amysamy1611 TutorTask226_Spring2025_Altair_Data_Visualization (#321) ( 9 hours ago) Mon May 19 23:05:15 2025 * e53f4c3d4 Indro TutorTask394_DataLoader_for_Historical_Causal_AutoML_and_FRED (#404) ( 7 days ago) Tue May 13 13:20:08 2025 (origin/fix_after_pull, origin/Spring2025_Real_Time_Bitcoin_Data_Analysis_with_Presto_461, origin/149, origin/147, origin/105) # Platform info system=Linux node name=e7240f5016eb release=5.15.0-1083-aws version=#90~20.04.1-Ubuntu SMP Tue Apr 22 09:59:53 UTC 2025 machine=x86_64 processor=x86_64 # psutils info cpu count=8 cpu freq=scpufreq(current=2499.9959999999996, min=0.0, max=0.0) memory=svmem(total=33280221184, available=27391090688, percent=17.7, used=5407014912, free=7352475648, active=7944695808, inactive=14081511424, buffers=2917085184, cached=17603645440, shared=1597440, slab=3444695040) disk usage=sdiskusage(total=218506772480, used=110164983808, free=108325011456, percent=50.4) # Docker info has_docker=True docker_version='28.1.1' docker_needs_sudo=False has_privileged_mode=True is_inside_docker=True has_sibling_containers_support=True has_docker_dind_support=True # Packages python: 3.12.3 cvxopt: ? cvxpy: ? gluonnlp: ? gluonts: ? joblib: 1.5.0 mxnet: ? numpy: 2.2.5 pandas: 2.2.3 pyarrow: 20.0.0 scipy: 1.15.3 seaborn: 0.13.2 sklearn: 1.6.1 statsmodels: 0.14.4 # Env vars CSFY_AWS_ACCESS_KEY_ID=empty CSFY_AWS_DEFAULT_REGION='' CSFY_AWS_S3_BUCKET='cryptokaizen-data' CSFY_AWS_SECRET_ACCESS_KEY=empty CSFY_AWS_SESSION_TOKEN='' CSFY_CI='' CSFY_ECR_BASE_PATH='623860924167.dkr.ecr.eu-north-1.amazonaws.com' CSFY_ENABLE_DIND='1' CSFY_FORCE_TEST_FAIL='' CSFY_HOST_NAME='dev2' CSFY_HOST_OS_NAME='Linux' CSFY_HOST_OS_VERSION='5.15.0-1083-aws' CSFY_HOST_USER_NAME='sonyan' CSFY_REPO_CONFIG_CHECK='True' CSFY_REPO_CONFIG_PATH='' GH_ACTION_ACCESS_TOKEN=empty PATH=.:./.git:./helpers_root:./docker_common:./causal_automl:./tutorial_tensorflow:./tutorial_pymc:./llms:./dev_scripts_tutorials:./UMD_MSML610:./tutorial_github_data605_style:./tutorial_langchain:./.mypy_cache:./tutorial_openai:./docs:./tutorial_github_causify_style:./tutorial_prophet:./devops:./sorrentum:./tutorial_data_science:./DATA605:./tutorial_neo4j:./tutorial_asana:./.github::/app:/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PYTHONPATH=/app:/app/helpers_root: # entrypoint.sh: './helpers_root/linters/base.py --files causal_automl/download_fred_data.py --num_threads serial' 04:23:11 - INFO hdbg.py init_logger:1018 > cmd='./helpers_root/linters/base.py --files causal_automl/download_fred_data.py --num_threads serial' 04:23:12 - INFO base.py _run_linter:403 Using num_threads='serial' since there is only one file to lint 04:23:12 - INFO base.py _lint:365 Linting file: 'causal_automl/download_fred_data.py' //////////////////////////////////////////////////////////////////////////////// ./linter_warnings.txt //////////////////////////////////////////////////////////////////////////////// cmd line='./helpers_root/linters/base.py --files causal_automl/download_fred_data.py --num_threads serial' file_paths=1 ['causal_automl/download_fred_data.py'] actions=25 ['add_python_init_files', 'add_toc_to_notebook', 'fix_md_links', 'lint_md', 'check_md_toc_headers', 'autoflake', 'fix_whitespaces', 'doc_formatter', 'isort', 'class_method_order', 'normalize_imports', 'format_separating_line', 'add_class_frames', 'remove_empty_lines_in_function', 'black', 'process_jupytext', 'check_file_size', 'check_filename', 'check_merge_conflict', 'check_import', 'warn_incorrectly_formatted_todo', 'check_md_reference', 'flake8', 'pylint', 'mypy'] //////////////////////////////////////////////////////////////////////////////// causal_automl/download_fred_data.py:105: [W0718(broad-exception-caught), FredDataDownloader.download_series] Catching too general exception Exception [pylint] causal_automl/download_fred_data.py:12: error: Cannot find implementation or library stub for module named "fredapi" [import-not-found] [mypy] causal_automl/download_fred_data.py:15: error: Cannot find implementation or library stub for module named "ratelimit" [import-not-found] [mypy]

Small update:
When I run the formatter part of linter locally (e.g., indro$ python helpers_root/linters/base.py -f causal_automl/download_fred_data.py --only_format), it adds the baptizing statement fine. I don't quite understand why it works there but not through i lint --files=... ( I tried deleting and reinstalling the docker image as well), but the following flow seems to be a decent workaround until I find a more permanent fix:

run indro$ python helpers_root/linters/base.py -f <file_name> --only_format for the formatting changes

run i lint --files=... for the warnings to address

WDYT?

Whatever works is fine as a workaround.
Just as a thought for further debugging -- you could try to play around with the code in linters/base.py and manually add some printing/logging or comment out some code to see what happens. E.g. comment out all actions besides normalize_import (that's the one that inserts the baptizing statement) and run Linter with/without the --only_format flag, and see if the baptizing statement is added. If so, then return the other actions one by one, and figure out the exact moment when the problem occurs.

Any hack is ok for now.

Yes, debugging is the way to go. It's worth reading the Linter documentation + the linter code base

Process-wise: This should have been an issue (and we should have solved it during the on-boarding phase).
Let's turn it into an issue so I can look into it (we can PP in our weekly sync)

…sal_AutoML_and_FRED

sonniki

Some small things left

Co-authored-by: Indrayudd Roy Chowdhury <indro@Indrayudds-MacBook-Air.local> Co-authored-by: sonniki <son.nik@mail.ru>

Indrayudd Roy Chowdhury added 3 commits May 7, 2025 04:08

TutorTask394: PR1 -> FredDataLoader that returns DF

6fe2128

TutorTask394: Pushing a duplicate file to later delete it to debug a …

7b1f5c4

…git discrepancy. Please ignore

TutorTask394: Remove duplicate

50ebad7

indrayudd requested a review from sonniki May 7, 2025 08:28

indrayudd self-assigned this May 7, 2025

indrayudd added the PR_for_reviewers The PR needs to be reviewed by RPs label May 7, 2025

sonniki suggested changes May 7, 2025

View reviewed changes

sonniki added PR_for_authors The PR needs changes and removed PR_for_reviewers The PR needs to be reviewed by RPs labels May 7, 2025

sonniki requested review from Shaunak01 and gpsaggese May 7, 2025 10:31

gpsaggese reviewed May 8, 2025

View reviewed changes

Comment thread causal_automl/fred_utils.py Outdated

Comment thread causal_automl/fred_utils.py Outdated

Comment thread causal_automl/fred_utils.py Outdated

gpsaggese mentioned this pull request May 8, 2025

Collect common review comment to automate causify-ai/helpers#685

Closed

Indrayudd Roy Chowdhury added 3 commits May 8, 2025 21:02

Reviewer Changes

6d21bf7

TutorTask394: Remove fred_utils.py

de1d24b

TutorTask394: Reviewer Changes again

2dab946

indrayudd requested a review from sonniki May 9, 2025 01:22

Merge branch 'master' into TutorTask394_DataLoader_for_Historical_Cau…

9fcefeb

…sal_AutoML_and_FRED

indrayudd added PR_for_reviewers The PR needs to be reviewed by RPs and removed PR_for_authors The PR needs changes labels May 9, 2025

sonniki requested a review from gpsaggese May 9, 2025 21:04

sonniki suggested changes May 9, 2025

View reviewed changes

sonniki reviewed May 9, 2025

View reviewed changes

Indrayudd Roy Chowdhury and others added 3 commits May 9, 2025 19:25

TutorTask394: Reviewer changes.

68f0f0b

TutorTask394: removed stray period.

dec20ea

Merge branch 'master' into TutorTask394_DataLoader_for_Historical_Cau…

c7b2c70

…sal_AutoML_and_FRED

indrayudd requested a review from sonniki May 9, 2025 23:33

sonniki suggested changes May 12, 2025

View reviewed changes

Comment thread causal_automl/download_fred_data.py Outdated

Comment thread causal_automl/download_fred_data.py Outdated

Comment thread causal_automl/download_fred_data.py Outdated

Comment thread causal_automl/download_fred_data.py

Comment thread causal_automl/download_fred_data.py Outdated

sonniki mentioned this pull request May 12, 2025

DataLoader for Historical Causal AutoML and FRED #394

Closed

sonniki removed the PR_for_reviewers The PR needs to be reviewed by RPs label May 12, 2025

sonniki added the PR_for_authors The PR needs changes label May 12, 2025

TutorTask394: Reviewer Changes

04b06c0

indrayudd requested a review from sonniki May 12, 2025 17:59

indrayudd added PR_for_reviewers The PR needs to be reviewed by RPs and removed PR_for_authors The PR needs changes labels May 12, 2025

sonniki reviewed May 12, 2025

View reviewed changes

Comment thread causal_automl/download_fred_data.py Outdated

Indrayudd Roy Chowdhury and others added 2 commits May 12, 2025 17:57

TutorTask394: Debugging commit

cd34a34

TutorTask394: Lint and improve comments

73e15eb

sonniki approved these changes May 13, 2025

View reviewed changes

sonniki merged commit e53f4c3 into master May 13, 2025
1 check passed

sonniki deleted the TutorTask394_DataLoader_for_Historical_Causal_AutoML_and_FRED branch May 13, 2025 13:20

vishwaksena-dingari pushed a commit that referenced this pull request May 16, 2025

TutorTask394_DataLoader_for_Historical_Causal_AutoML_and_FRED (#404)

920cef1

Co-authored-by: Indrayudd Roy Chowdhury <indro@Indrayudds-MacBook-Air.local> Co-authored-by: sonniki <son.nik@mail.ru>

gpsaggese mentioned this pull request May 23, 2025

Linter issue in indro's setup causify-ai/helpers#755

Closed

Conversation

indrayudd commented May 7, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

indrayudd May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonniki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

indrayudd May 12, 2025 •

edited

Loading