Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Experiment D #4

Closed
wants to merge 4 commits into from
Closed

Experiment D #4

wants to merge 4 commits into from

Conversation

b-long
Copy link
Owner

@b-long b-long commented Sep 25, 2022 •
edited
Loading

Build Results

  • Succeeds: Yes
  • Fast? Faster than experiments A-C

Flows working?

  • basic-add-flow: Yes
  • basic-pandas-flow: No
    • ModuleNotFoundError: No module named 'numpy'

Flow run log:

25 September 2022,02:24:58 	agent	INFO	Submitted for execution: Container ID: 525ccda2d9011c6696300170d8c4ad780b00270c873810c627a99c6aef77ac5d
25 September 2022,02:25:02 	prefect.CloudFlowRunner	INFO	Beginning Flow run for 'basic-pandas-flow'
25 September 2022,02:25:02 	prefect.CloudTaskRunner	INFO	Task 'pandas_task': Starting task run...
25 September 2022,02:25:02 	prefect.pandas_task	INFO	************Doing pandas thing for FOO='foo'************
25 September 2022,02:25:02 	prefect.CloudTaskRunner	ERROR	Task 'pandas_task': Exception encountered during task execution!
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine/task_runner.py", line 880, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/executors.py", line 468, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/Users/b-long/Desktop/github/b-long/aid/aid/prefect/basic_prefect_with_pandas.py", line 16, in pandas_task
  File "/workdir/aid/etl/arbitrary_pandas.py", line 5, in do_pandas
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
25 September 2022,02:25:02 	prefect.CloudTaskRunner	INFO	Task 'pandas_task': Finished task run for task with final state: 'Failed'
25 September 2022,02:25:02 	prefect.CloudFlowRunner	INFO	Flow run FAILED: some reference tasks failed.

Full build log:

$ ./aid/prefect/step-3-register-flow.sh
Configuration file exists at /Users/b-long/Library/Application Support/pypoetry, reusing this directory.

Consider moving configuration to /Users/b-long/Library/Preferences/pypoetry, as support for the legacy directory will be removed in an upcoming release.
/Users/b-long/Desktop/github/b-long//aid/.venv/lib/python3.10/site-packages/prefect/storage/docker.py:325: UserWarning: This Docker storage object has no `registry_url`, and will not be pushed.
  self._build_image(push=push)
[2022-09-25 14:11:41-0400] INFO - prefect.Docker | Building the flow's Docker storage...
Step 1/14 : FROM prefecthq/prefect:1.3.1-python3.10 as python-base
 ---> 16d2aaf55962
Step 2/14 : RUN pip install poetry==1.2.1
 ---> Running in cced7b293308
Collecting poetry==1.2.1
  Downloading poetry-1.2.1-py3-none-any.whl (211 kB)
Collecting crashtest<0.4.0,>=0.3.0
  Downloading crashtest-0.3.1-py3-none-any.whl (7.0 kB)
Collecting poetry-core==1.2.0
  Downloading poetry_core-1.2.0-py3-none-any.whl (525 kB)
Collecting jsonschema<5.0.0,>=4.10.0
  Downloading jsonschema-4.16.0-py3-none-any.whl (83 kB)
Collecting poetry-plugin-export<2.0.0,>=1.0.7
  Downloading poetry_plugin_export-1.0.7-py3-none-any.whl (9.8 kB)
Collecting cleo<2.0.0,>=1.0.0a5
  Downloading cleo-1.0.0a5-py3-none-any.whl (78 kB)
Collecting cachy<0.4.0,>=0.3.0
  Downloading cachy-0.3.0-py2.py3-none-any.whl (20 kB)
Collecting cachecontrol[filecache]<0.13.0,>=0.12.9
  Downloading CacheControl-0.12.11-py2.py3-none-any.whl (21 kB)
Collecting shellingham<2.0,>=1.5
  Downloading shellingham-1.5.0-py2.py3-none-any.whl (9.3 kB)
Requirement already satisfied: requests-toolbelt<0.10.0,>=0.9.1 in /usr/local/lib/python3.10/site-packages (from poetry==1.2.1) (0.9.1)
Collecting keyring>=21.2.0
  Downloading keyring-23.9.3-py3-none-any.whl (35 kB)
Requirement already satisfied: dulwich<0.21.0,>=0.20.46 in /usr/local/lib/python3.10/site-packages (from poetry==1.2.1) (0.20.46)
Collecting html5lib<2.0,>=1.0
  Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting platformdirs<3.0.0,>=2.5.2
  Downloading platformdirs-2.5.2-py3-none-any.whl (14 kB)
Requirement already satisfied: urllib3<2.0.0,>=1.26.0 in /usr/local/lib/python3.10/site-packages (from poetry==1.2.1) (1.26.12)
Collecting tomlkit!=0.11.2,!=0.11.3,<1.0.0,>=0.11.1
  Downloading tomlkit-0.11.4-py3-none-any.whl (35 kB)
Collecting pkginfo<2.0,>=1.5
  Downloading pkginfo-1.8.3-py2.py3-none-any.whl (26 kB)
Collecting virtualenv!=20.4.5,!=20.4.6,>=20.4.3
  Downloading virtualenv-20.16.5-py3-none-any.whl (8.8 MB)
Requirement already satisfied: packaging>=20.4 in /usr/local/lib/python3.10/site-packages (from poetry==1.2.1) (21.3)
Collecting pexpect<5.0.0,>=4.7.0
  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Requirement already satisfied: requests<3.0,>=2.18 in /usr/local/lib/python3.10/site-packages (from poetry==1.2.1) (2.28.1)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
  Downloading pyrsistent-0.18.1.tar.gz (100 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting attrs>=17.4.0
  Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting pylev<2.0.0,>=1.3.0
  Downloading pylev-1.4.0-py2.py3-none-any.whl (6.1 kB)
Requirement already satisfied: msgpack>=0.5.2 in /usr/local/lib/python3.10/site-packages (from cachecontrol[filecache]<0.13.0,>=0.12.9->poetry==1.2.1) (1.0.4)
Collecting lockfile>=0.9; extra == "filecache"
  Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
Collecting jeepney>=0.4.2; sys_platform == "linux"
  Downloading jeepney-0.8.0-py3-none-any.whl (48 kB)
Collecting SecretStorage>=3.2; sys_platform == "linux"
  Downloading SecretStorage-3.3.3-py3-none-any.whl (15 kB)
Collecting jaraco.classes
  Downloading jaraco.classes-3.2.2-py3-none-any.whl (6.0 kB)
Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.10/site-packages (from html5lib<2.0,>=1.0->poetry==1.2.1) (1.16.0)
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting distlib<1,>=0.3.5
  Downloading distlib-0.3.6-py2.py3-none-any.whl (468 kB)
Collecting filelock<4,>=3.4.1
  Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.10/site-packages (from packaging>=20.4->poetry==1.2.1) (3.0.9)
Collecting ptyprocess>=0.5
  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/site-packages (from requests<3.0,>=2.18->poetry==1.2.1) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/site-packages (from requests<3.0,>=2.18->poetry==1.2.1) (2022.6.15)
Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.10/site-packages (from requests<3.0,>=2.18->poetry==1.2.1) (2.1.1)
Requirement already satisfied: cryptography>=2.0 in /usr/local/lib/python3.10/site-packages (from SecretStorage>=3.2; sys_platform == "linux"->keyring>=21.2.0->poetry==1.2.1) (38.0.1)
Collecting more-itertools
  Downloading more_itertools-8.14.0-py3-none-any.whl (52 kB)
Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/site-packages (from cryptography>=2.0->SecretStorage>=3.2; sys_platform == "linux"->keyring>=21.2.0->poetry==1.2.1) (1.15.1)
Requirement already satisfied: pycparser in /usr/local/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.0->SecretStorage>=3.2; sys_platform == "linux"->keyring>=21.2.0->poetry==1.2.1) (2.21)
Building wheels for collected packages: pyrsistent
  Building wheel for pyrsistent (PEP 517): started
  Building wheel for pyrsistent (PEP 517): finished with status 'done'
  Created wheel for pyrsistent: filename=pyrsistent-0.18.1-cp310-cp310-linux_x86_64.whl size=71441 sha256=0d6c6a55f2fc171214d43bf4b8764022e87cc578ace71de681af7ddc20328868
  Stored in directory: /root/.cache/pip/wheels/42/cf/03/ae35459d6c32fce920e7eb28fa325fdeced7a39fb9b98c56f0
Successfully built pyrsistent
Installing collected packages: crashtest, poetry-core, pyrsistent, attrs, jsonschema, poetry-plugin-export, pylev, cleo, cachy, lockfile, cachecontrol, shellingham, jeepney, SecretStorage, more-itertools, jaraco.classes, keyring, webencodings, html5lib, platformdirs, tomlkit, pkginfo, distlib, filelock, virtualenv, ptyprocess, pexpect, poetry
Successfully installed SecretStorage-3.3.3 attrs-22.1.0 cachecontrol-0.12.11 cachy-0.3.0 cleo-1.0.0a5 crashtest-0.3.1 distlib-0.3.6 filelock-3.8.0 html5lib-1.1 jaraco.classes-3.2.2 jeepney-0.8.0 jsonschema-4.16.0 keyring-23.9.3 lockfile-0.12.2 more-itertools-8.14.0 pexpect-4.8.0 pkginfo-1.8.3 platformdirs-2.5.2 poetry-1.2.1 poetry-core-1.2.0 poetry-plugin-export-1.0.7 ptyprocess-0.7.0 pylev-1.4.0 pyrsistent-0.18.1 shellingham-1.5.0 tomlkit-0.11.4 virtualenv-20.16.5 webencodings-0.5.1
WARNING: You are using pip version 20.2.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

Removing intermediate container cced7b293308
 ---> 2bdfb063dead
Step 3/14 : COPY pyproject.toml poetry.lock aid /opt/aid
When using COPY with more than one source file, the destination must be a directory and end with a /
Flow URL: http://localhost:8080/aid-tenant/flow/afd95d87-ba5b-4048-b547-9eba85e4d715
 └── ID: e57402ce-1b91-40f0-b59c-fb6618cce300
 └── Project: aid-tenant
 └── Labels: []
(aid-py3.10) b-longs-MBP:aid b-long$ ./aid/prefect/step-3-register-flow.sh
Configuration file exists at /Users/b-long/Library/Application Support/pypoetry, reusing this directory.

Consider moving configuration to /Users/b-long/Library/Preferences/pypoetry, as support for the legacy directory will be removed in an upcoming release.
/Users/b-long/Desktop/github/b-long//aid/.venv/lib/python3.10/site-packages/prefect/storage/docker.py:325: UserWarning: This Docker storage object has no `registry_url`, and will not be pushed.
  self._build_image(push=push)
[2022-09-25 14:18:59-0400] INFO - prefect.Docker | Building the flow's Docker storage...
Step 1/14 : FROM prefecthq/prefect:1.3.1-python3.10 as python-base
 ---> 16d2aaf55962
Step 2/14 : RUN pip install poetry==1.2.1
 ---> Using cache
 ---> 2bdfb063dead
Step 3/14 : COPY pyproject.toml poetry.lock aid /opt/aid
When using COPY with more than one source file, the destination must be a directory and end with a /
Flow URL: http://localhost:8080/aid-tenant/flow/1dd5584d-36da-4a09-8d91-dd498c1af316
 └── ID: f66bc44c-9ccb-4bfd-afbe-f3075f6e706d
 └── Project: aid-tenant
 └── Labels: []

@b-long
Copy link
Owner Author

b-long commented Oct 29, 2022

Superseded by #5 .

@b-long b-long closed this Oct 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@b-long