Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make video upload to S3 compatible with SigV4 authentication #22080

Open
wants to merge 1 commit into
base: master
from

Conversation

@regisb
Copy link
Contributor

regisb commented Oct 18, 2019

The studio has a feature that allows course teams to upload videos
directly to S3. This is a very convenient feature, but it works only for
some S3 regions. The reason is that some regions require SigV4
authentication. It is possible to enable SigV4 authentication in boto by
setting the S3_USE_SIGV4 environment variable. However, this requires
the "host" setting to be passed to the boto.connection.Connection
constructor.

Actually, there are many other settings that should be passed to the
boto constructor, and this is precisely what the
storages.backends.s3.S3BotoStorage constructor does. So we replace the
call to boto by a call to the storages app.

All other calls to boto are made via the storages app in edx-platform.
As far as we know, the only exception left is in ORA2.

edX.org has not faced this problem because they (probably) use a
different authentication capability.

This change is important because it allows video uploads to, for
instance, the us-east-2 region. The S3 emulator called MinIO (minio.io)
also requires SigV4. This emulator is useful for bringing object storage
capabilities to Open edX administrators which may not have access to
AWS.

@edx-webhook

This comment has been minimized.

Copy link

edx-webhook commented Oct 18, 2019

Thanks for the pull request, @regisb! I've created OSPR-3889 to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams.

Feel free to add as much of the following information to the ticket:

  • supporting documentation
  • edx-code email threads
  • timeline information ("this must be merged by XX date", and why that is)
  • partner information ("this is a course on edx.org")
  • any other information that can help Product understand the context for the PR

All technical communication about the code itself will still be done via the GitHub pull request interface. As a reminder, our process documentation is here.

regisb added a commit to overhangio/tutor-minio that referenced this pull request Oct 18, 2019
This requires that SigV4 is enabled:
edx/edx-platform#22080

Also, bump released version.
@natabene

This comment has been minimized.

Copy link
Member

natabene commented Oct 18, 2019

@regisb Thank you for your contribution. Please let me know once it is ready for our review.

@regisb regisb force-pushed the regisb:regisb/video-s3-sigv4 branch 3 times, most recently from c47c63c to 12884fc Oct 24, 2019
@regisb

This comment has been minimized.

Copy link
Contributor Author

regisb commented Oct 24, 2019

I believe the jenkins pipeline failed for a different reason. Maybe restart?

@natabene

This comment has been minimized.

Copy link
Member

natabene commented Oct 24, 2019

jenkins run all

The studio has a feature that allows course teams to upload videos
directly to S3. This is a very convenient feature, but it works only for
some S3 regions. The reason is that some regions require SigV4
authentication. It is possible to enable SigV4 authentication in boto by
setting the S3_USE_SIGV4 environment variable. However, this requires
the "host" setting to be passed to the boto.connection.Connection
constructor.

Actually, there are many other settings that should be passed to the
boto constructor, and this is precisely what the
storages.backends.s3.S3BotoStorage constructor does. So we replace the
call to boto by a call to the storages app.

All other calls to boto are made via the storages app in edx-platform.
As far as we know, the only exception left is in ORA2.

edX.org has not faced this problem because they (probably) use a
different authentication capability.

This change is important because it allows video uploads to, for
instance, the us-east-2 region.  The S3 emulator called MinIO (minio.io)
also requires SigV4. This emulator is useful for bringing object storage
capabilities to Open edX administrators which may not have access to
AWS.
@regisb regisb force-pushed the regisb:regisb/video-s3-sigv4 branch from 12884fc to ef55cdc Jan 9, 2020
@edx-status-bot

This comment has been minimized.

Copy link

edx-status-bot commented Jan 9, 2020

Your PR has finished running tests. There were no failures.

@natabene

This comment has been minimized.

Copy link
Member

natabene commented Jan 10, 2020

@regisb Lining this up for a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.