ImgUploader: add support for non-amazon S3 #20354

emmericp · 2019-11-13T11:55:30Z

Make hostname for S3 upload configurable to support non-Amazon S3 backends. Also add an option to use path style access as some poorly configured S3 backends sometimes don't support the default subdomain mode.

Fixes #8157

Question: how is the S3 backend usually tested? I found an integration test but no information on how it is usually run?

fixes grafana#11240

marefr · 2019-11-13T18:53:44Z

@emmericp Thanks. The issue you're referring to is closed. However, there's another one #8157. Please update PR description if you think your PR solves this feature request taking this #8157 (comment) into account.

emmericp · 2019-11-13T19:48:34Z

Oh, I didn't see that one. Yeah, it should work with virtually anything that implements the bare minimum of S3 because it doesn't use any advanced feature at all. I've only tested it against Ceph/radosgw which implements most S3 and most S3-compatible services you find on the Internet run it.

Updated

stale · 2019-12-02T03:18:38Z

This pull request has been automatically marked as stale because it has not had activity in the last 2 weeks. It will be closed in 30 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

emmericp · 2019-12-02T09:23:35Z

ping

stale · 2019-12-16T10:09:00Z

This pull request has been automatically marked as stale because it has not had activity in the last 2 weeks. It will be closed in 30 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

emmericp · 2019-12-16T10:51:26Z

ping. happy to provide test access to a public S3 bucket

marefr · 2019-12-16T13:19:56Z

We're planning to review this soon. Please have patient. Thanks

stale · 2019-12-31T13:53:03Z

This pull request has been automatically marked as stale because it has not had activity in the last 2 weeks. It will be closed in 30 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

aknuds1

LGTM!

Please note though that I renamed the 'host' setting to 'endpoint', in order to be consistent with S3 nomenclature.

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

aknuds1 · 2020-01-02T14:11:55Z

Merged - thanks for contributing to Grafana!

* imguploader: add support for non-Amazon S3 endpoints and forcing of path-style S3 addressing fixes grafana#11240 Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>

IamAliBaba · 2020-12-09T09:21:21Z

Hello,
We are testing Ceph S3 capability with Grafana seems not working or no clean documentation.

"Failed to render and upload alert panel image." logger=alerting.notifier ruleId=7 error="Could not find bucket setting for image.uploader.s3"

Here is my config.

[external_image_storage.s3]
endpoint = s3.xxx.com
;path_style_access = https://s3.xxxx.com/BUCKET/
;bucket = xx-grafana-bucket
;region = us-east-1
;path =
bucket_url = https://s3.xxxx.com/xx-grafana-bucket
access_key = xxx
secret_key = xxx

Can you please give a more config info.

Thanks.

IamAliBaba · 2020-12-09T10:36:40Z

bellow info show that S3 works from that server .

root@dash:~# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key [xxxxxx]:
Secret Key [xxxxx]:
Default Region [us]: us

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.xxxxx.com]:

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [s3.xxxxx.com/(bucket)s]:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: xxxxx
Secret Key: xxxxx
Default Region: us
S3 Endpoint: s3.xxxxx.com
DNS-style bucket+hostname:port template for accessing a bucket: s3.xxxx.com/(bucket)s
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

IamAliBaba · 2020-12-09T12:32:00Z

pkg/components/imguploader/imguploader.go

@@ -35,7 +35,7 @@ func NewImageUploader() (ImageUploader, error) {
 			return nil, err
 		}

-		host := s3sec.Key("host").MustString("")
+		endpoint := s3sec.Key("endpoint").MustString("")
 		pathStyleAccess := s3sec.Key("path_style_access").MustBool(false)


this is not working line.

IamAliBaba · 2020-12-09T12:46:52Z

Hi ,
We have non-AWS S3 and having issue with ImageUploader .

our S3 path_style_access is https://s3.xxxxxxx.com/bucket/key not https://bucket.s3.xxxxxxx.com/key .

I have this configuration on grafana.ini

[external_image_storage.s3]
endpoint = https://s3.XXXXXX.com
path_style_access = https://s3.XXXXX.com/BUCKET/KEY
bucket = XX-grafana-bucket
region = us
;path =
access_key = XXXXX
secret_key = XXXXX

i changed path_style_access to our custom but it still trying to connect to this one ` https://bucket.s3.xxxxxxx.com/key

LOG:

t=2020-12-09T07:16:37-0500 lvl=eror msg="Failed to render and upload alert panel image." logger=alerting.notifier ruleId=7 error="RequestError: send request failed\ncaused by: Put "**https://xxx-grafana-bucket.s3.xxxxxx.com/gSvYnzqm8MDUnpMyIk7X.png**\": x509: certificate is valid for *.xxxxxxxx.com, xxxxxxxx.com, not xx-grafana-bucket.s3.xxxxxxx.com"

Seems BUG.

oddlittlebird · 2020-12-09T21:54:11Z

Hi @IamAliBaba! I'm sorry you are having trouble with this. Please file a bug report.

imguploader: add support for non-amazon S3

63bb5fc

fixes grafana#11240

emmericp force-pushed the 3rd-party-s3-support branch from 6807df8 to 63bb5fc Compare November 13, 2019 12:09

marefr added area/backend pr/external This PR is from external contributor labels Nov 13, 2019

add docs

ed221e4

stale bot added the stale Issue with no recent activity label Dec 2, 2019

stale bot removed the stale Issue with no recent activity label Dec 2, 2019

stale bot added the stale Issue with no recent activity label Dec 16, 2019

stale bot removed the stale Issue with no recent activity label Dec 16, 2019

marefr requested review from a team, kylebrandt, marcusolsson and aknuds1 and removed request for a team, kylebrandt and marcusolsson December 17, 2019 10:37

aknuds1 self-assigned this Dec 17, 2019

aknuds1 added this to To do in Backend Platform Squad Dec 17, 2019

stale bot added the stale Issue with no recent activity label Dec 31, 2019

Merge remote-tracking branch 'origin/master' into 3rd-party-s3-support

b3102b2

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

emmericp requested a review from oddlittlebird as a code owner January 2, 2020 08:14

stale bot removed the stale Issue with no recent activity label Jan 2, 2020

vendor/: Sync vendored package

8531fcf

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

aknuds1 moved this from To do to Under review in Backend Platform Squad Jan 2, 2020

s3uploader: Use term 'endpoint' instead of 'host'

57116c1

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

aknuds1 approved these changes Jan 2, 2020

View reviewed changes

aknuds1 added 2 commits January 2, 2020 13:04

Update conf/sample.ini

0b156d2

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

Docs: Improve S3 documentation

efdec85

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

aknuds1 merged commit 42032f6 into grafana:master Jan 2, 2020

Backend Platform Squad automation moved this from Under review to Done Jan 2, 2020

marefr added this to the 6.6.0-beta1 milestone Jan 28, 2020

marefr mentioned this pull request Jan 28, 2020

[Bug] Minio as external image storage #8156

Closed

IamAliBaba reviewed Dec 9, 2020

View reviewed changes

grafana locked as resolved and limited conversation to collaborators Dec 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ImgUploader: add support for non-amazon S3 #20354

ImgUploader: add support for non-amazon S3 #20354

emmericp commented Nov 13, 2019 •

edited

marefr commented Nov 13, 2019

emmericp commented Nov 13, 2019

stale bot commented Dec 2, 2019

emmericp commented Dec 2, 2019

stale bot commented Dec 16, 2019

emmericp commented Dec 16, 2019

marefr commented Dec 16, 2019

stale bot commented Dec 31, 2019

aknuds1 left a comment

aknuds1 commented Jan 2, 2020

IamAliBaba commented Dec 9, 2020

IamAliBaba commented Dec 9, 2020

IamAliBaba Dec 9, 2020

IamAliBaba commented Dec 9, 2020

oddlittlebird commented Dec 9, 2020 •

edited

ImgUploader: add support for non-amazon S3 #20354

ImgUploader: add support for non-amazon S3 #20354

Conversation

emmericp commented Nov 13, 2019 • edited

marefr commented Nov 13, 2019

emmericp commented Nov 13, 2019

stale bot commented Dec 2, 2019

emmericp commented Dec 2, 2019

stale bot commented Dec 16, 2019

emmericp commented Dec 16, 2019

marefr commented Dec 16, 2019

stale bot commented Dec 31, 2019

aknuds1 left a comment

Choose a reason for hiding this comment

aknuds1 commented Jan 2, 2020

IamAliBaba commented Dec 9, 2020

IamAliBaba commented Dec 9, 2020

IamAliBaba Dec 9, 2020

Choose a reason for hiding this comment

IamAliBaba commented Dec 9, 2020

oddlittlebird commented Dec 9, 2020 • edited

emmericp commented Nov 13, 2019 •

edited

oddlittlebird commented Dec 9, 2020 •

edited