Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make S3 support optional #1252

Merged
merged 4 commits into from Apr 27, 2021
Merged

Make S3 support optional #1252

merged 4 commits into from Apr 27, 2021

Conversation

pquentin
Copy link
Member

@pquentin pquentin commented Apr 25, 2021

Closes #1095.

With this pull request, boto3 is no longer installed by default but should be installed using python -m pip install esrally[s3] instead.

When a S3 base-url is provided, here's what users see:

[INFO] Preparing for race ...
[ERROR] S3 support is optional. Install it with `python -m pip install esrally[s3]`

[ERROR] Cannot race. Error in track preparator
        No module named 'boto3'

...

So, we let the ImportError propagate, but also explain how to get S3 support.

To test this manually in an existing virtual environment, uninstalling boto3 manually is not enough, as pkg_resource.require("esrally") will complain that it's missing. Recreating the virtual environment from scratch and removing any .egg-info files should fix this.

For what it's worth, I haven't managed to get the integration tests to work locally, but they don't seem to use S3, so they probably still pass.

(This is only about S3 as suggested by #1095, but I'll be happy to do the same thing for GCS if this PR gets merged.)

@@ -70,10 +70,22 @@ def finish(self):
self.p.finish()


def _fake_import_boto3():
# This function only exists to be mocked in tests to raise an ImportError, in
# order to simulate the absence of boto3
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is quite ugly, but the alternative is to use del sys.modules["boto3"], which is an ugly and global hack, and likely to break things. While we use this hack succesfully in urllib3 to test support for Python versions without ssl, this:

So I would suggest avoiding touching sys.modules.

@dliappis dliappis requested a review from DJRickyB April 26, 2021 06:51
@DJRickyB DJRickyB added :Packaging Installation issues or packaging problems :Usability Makes Rally easier to use enhancement Improves the status quo labels Apr 27, 2021
@DJRickyB DJRickyB added this to the 2.2.1 milestone Apr 27, 2021
@DJRickyB
Copy link
Contributor

@elasticmachine run tests please

@pquentin
Copy link
Member Author

Thanks for running the tests! I'll fix the lint

@pquentin
Copy link
Member Author

@DJRickyB Can you please run the tests again? I guess @elasticmachine won't listen if I try to ask myself

@DJRickyB
Copy link
Contributor

@elasticmachine run tests

Copy link
Contributor

@DJRickyB DJRickyB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pquentin thanks a ton! I have one remaining requirement before approval: please preserve the behavior where make install ends up installing boto3+dependencies. for our purposes I think it's ok if [s3] resolution occurs in either make install-user or a new make target altogether

@pquentin
Copy link
Member Author

@DJRickyB Thank you for the review! Done

Copy link
Contributor

@DJRickyB DJRickyB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thank you once again for your contribution

@DJRickyB DJRickyB merged commit 51754b7 into elastic:master Apr 27, 2021
@pquentin
Copy link
Member Author

Thank you for the quick reviews! I'll work on making GCS support optional next.

@DJRickyB
Copy link
Contributor

@pquentin thank you, but we are looking at keeping GCS support first-class for the time being

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improves the status quo :Packaging Installation issues or packaging problems :Usability Makes Rally easier to use
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make S3 dependency in Rally optional
2 participants