Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deployment to s3-bucket: PosixPath error #7583

Closed
4 tasks done
mjkanji opened this issue Nov 18, 2022 · 1 comment · Fixed by PrefectHQ/prefect-aws#169
Closed
4 tasks done

Deployment to s3-bucket: PosixPath error #7583

mjkanji opened this issue Nov 18, 2022 · 1 comment · Fixed by PrefectHQ/prefect-aws#169
Labels
bug Something isn't working

Comments

@mjkanji
Copy link

mjkanji commented Nov 18, 2022

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar issue and didn't find it.
  • I searched the Prefect documentation for this issue.
  • I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

I'm trying to follow the Storage and infrastructure tutorial in the docs and am unable to create a deployment with S3 as the storage block. When creating the deployment from the CLI, I get an error relating because the code is expecting a string (it uses the endswith method) but receiving a PosixPath. See Reproduction below for more details.

As an aside, the tutorial is also out-of-date and needs to be updated to change the storage block type from s3 to s3-bucket -- i.e., -sb s3-bucket/log-test instead of -sb s3/log-test, as this is what prefect block ls shows as the slug. I'm assuming this was a recent change to, maybe, the prefect-aws collection?

Reproduction

Follow the [Deployments](https://docs.prefect.io/tutorials/deployments/) and [Storage and infrastructure](https://docs.prefect.io/tutorials/storage/) tutorials. The command I'm having issues with is:


$ prefect deployment build ./log_flow.py:log_flow -n log-flow-s3 -sb s3/log-test -q test -o log-flow-s3-deployment.yaml

You'll also need to have the following files (see the Deployments tutorial):

log_flow.py

import sys
import prefect
from prefect import flow, task, get_run_logger
from utilities import AN_IMPORTED_MESSAGE


@task
def log_task(name):
    logger = get_run_logger()
    logger.info("Hello %s!", name)
    logger.info("Prefect Version = %s 🚀", prefect.__version__)
    logger.debug(AN_IMPORTED_MESSAGE)


@flow()
def log_flow(name: str):
    log_task(name)


if __name__ == "__main__":
    name = sys.argv[1]
    log_flow(name)

Other dependencies:

$ echo '{"some-piece-of-config": 100}' > config.json
$ echo 'AN_IMPORTED_MESSAGE = "Hello from another file"' > utilities.py

Error

Found flow 'log-flow'
Deployment YAML created at '/home/some_path/prefect/log-flow-s3-deployment.yaml'.
Traceback (most recent call last):
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 201, in coroutine_wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 152, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/site-packages/prefect/cli/deployment.py", line 902, in build
    f"Successfully uploaded {file_count} files to {deployment.location}",
  File "/home/me/mambaforge/envs/prefect/lib/python3.10/site-packages/prefect/deployments.py", line 300, in location
    if not self.storage.basepath.endswith("/")
AttributeError: 'PosixPath' object has no attribute 'endswith'

Versions

Version:             2.6.7
API version:         0.8.3
Python version:      3.10.6
Git commit:          9074c158
Built:               Thu, Nov 10, 2022 1:07 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         hosted

Additional context

This seems related to #7289.

@mjkanji mjkanji added bug Something isn't working status:triage labels Nov 18, 2022
@mjkanji mjkanji changed the title Deployment to S3 storage: Deployment to S3 storage: PosixPath error Nov 18, 2022
@mjkanji mjkanji changed the title Deployment to S3 storage: PosixPath error Deployment to s3-bucket: PosixPath error Nov 18, 2022
@lsetiawan
Copy link

I'm running into this issue also! I think this is caused by the fact that basepath field of S3Bucket model is a pathlib.Path rather than a regular string. See https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/s3.py#L272-L276.

The codeblock in deployments.py within prefect seem to be expecting a string when calling .basepath from self.storage:

if self.storage:
location = (
self.storage.basepath + "/"
if not self.storage.basepath.endswith("/")
else ""
)
I'm not sure if this is a bug on prefect side or prefect-aws. Is it expected that self.storage.basepath should always be a string?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants