Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a flag to control escape hatch log #1819

Merged
merged 2 commits into from
May 2, 2024
Merged

Conversation

wangchy27
Copy link
Collaborator

@wangchy27 wangchy27 commented May 2, 2024

We print a warning if a package is supposed to be escaped but not (e.g. User include the package in conda environment). However, the log may later disturb s3op wokers since the communication between subprocesses is depend on stdout.

For example:

  1. bdp-boto is suppposed to be escaped
  2. User specify a version of bdp-boto in their conda environment
  3. We print out "Not using environment escape for 'bdp_boto' as module present"
  4. User do a S3.put_many()
  5. The s3 option would fail since it doesn't understand the log "Not using....."
File "/root/metaflow/metaflow/plugins/datatools/s3/s3.py", line 1561, in _update_out_lines
>      out_lines[int(idx.decode(encoding="utf-8"))] = rest
>  ValueError: invalid literal for int() with base 10: 'Not'

Thus, we add a flag to control printing out these warning or not as a short term solution to the issue

@wangchy27 wangchy27 merged commit 0dc8cd6 into master May 2, 2024
26 checks passed
@wangchy27 wangchy27 deleted the fix/escape_hatch_log branch May 2, 2024 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants