Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using a CDN in front of S3 storage (Digital Ocean) avatar URL is malformed. #4041

Closed
BorisKourt opened this issue Dec 29, 2022 · 9 comments

Comments

@BorisKourt
Copy link

BorisKourt commented Dec 29, 2022

I realized that this is a separate issue, this original comment is at: #2866 (comment)


Single image posts work correctly with the CDN setup as AWS_URL (haven't tried galleries). But for profile pictures the image path is malformed:

https://example.comhttps//cdn.example.com/public/avatars/051/405/995/096/096/768/1/kYSiafmPTyUrB4Jsjjlq_avatar.png?v=2

It is adding https://example.com in front of the AWS_URL value of https//cdn.example.com

It should just be:

https//cdn.example.com/public/avatars/051/405/995/096/096/768/1/kYSiafmPTyUrB4Jsjjlq_avatar.png?v=2

So far I have only seen this for profile pictures, and not with any other uploads.

And I can see the file at the correct path in the bucket.

Where in the codebase is this URL formed?


Annotated storage configuration for reference:

## S3 Configuration (Post-Installer)
PF_ENABLE_CLOUD=true
FILESYSTEM_DRIVER=s3 # "spaces" driver doesn't work with Digital Ocean CDN
FILESYSTEM_CLOUD=s3
AWS_ACCESS_KEY_ID=<snip>
AWS_SECRET_ACCESS_KEY=<snip>
AWS_DEFAULT_REGION=us-east-1 # Doesn't have to be the DO region. This is an AWS region.
AWS_BUCKET=<snip>
AWS_ENDPOINT=https://ams3.digitaloceanspaces.com # Not the same endpoint as what DO links to in the bucket. Note the missing bucket name.
AWS_URL=https://your.cdn.subdomain.com # Since I am using the CDN this was the only way to do it. https:// is mandatory.

Update:

As per below this is the updated workaround config:

## S3 Configuration (Post-Installer)
PF_ENABLE_CLOUD=true
FILESYSTEM_DRIVER=local # <- Main change, allows avatars to function. Though served locally. Posts still come from S3
FILESYSTEM_CLOUD=s3
AWS_ACCESS_KEY_ID=<snip>
AWS_SECRET_ACCESS_KEY=<snip>
AWS_DEFAULT_REGION=us-east-1 # Doesn't have to be the DO region. This is an AWS region.
AWS_BUCKET=<snip>
AWS_ENDPOINT=https://ams3.digitaloceanspaces.com # Not the same endpoint as what DO links to in the bucket. Note the missing bucket name.
AWS_URL=https://your.cdn.subdomain.com # Since I am using the CDN this was the only way to do it. https:// is mandatory.
@okpierre
Copy link
Contributor

Could it be broken again? It was fixed according to @dansup

See: #3559

@BorisKourt
Copy link
Author

@okpierre it seems that the advice to disable AWS_URL there would break access via the CDN.

@okpierre
Copy link
Contributor

@BorisKourt also you need to set this FILESYSTEM_DRIVER=local

dansup mentioned this should not be s3

@BorisKourt
Copy link
Author

BorisKourt commented Dec 29, 2022

If I use the FILESYSTEM_DRIVER=local the avatars work correctly. But unlike regular posts they don't appear to switch to the S3 storage after a period of time. (Images uploaded to a post first serves from local storage then from S3, as is described elsewhere.)

This is not a big deal as the avatars aren't really slowing anything down or taking up too much space. But for migration and consistency could be helpful to adjust this behavior.

This all still works with AWS_URL set to the CDN.


As an aside I wonder how this two stage process is picked up by other Activity Pub nodes. Which URL they cache and re-reference.

@joho1968
Copy link

I think this is related to #4031

@mitexleo
Copy link

If I use the FILESYSTEM_DRIVER=local the avatars work correctly. But unlike regular posts they don't appear to switch to the S3 storage after a period of time. (Images uploaded to a post first serves from local storage then from S3, as is described elsewhere.)

This is not a big deal as the avatars aren't really slowing anything down or taking up too much space. But for migration and consistency could be helpful to adjust this behavior.

This all still works with AWS_URL set to the CDN.

As an aside I wonder how this two stage process is picked up by other Activity Pub nodes. Which URL they cache and re-reference.

You can store Avatars to s3 too ...
You have to set filesystem drive to local ... Otherwise it will break things

@mitexleo
Copy link

To store avatars to the s3 :
Add
PF_LOCAL_AVATAR_TO_CLOUD=true
In .env

Then run php artisan avatar:storage

@jdaviescoates
Copy link

php artisan avatar:storage

That doesn't seem to be defined anymore:

# php artisan avatar:storage

                                                                                                                        
 Command "avatar:storage" is not defined.                                                                               
                                                                                                                        

 Do you want to run "avatars:sync" instead?  (yes/no) [no]:
 > yes

Welcome to the avatar sync manager

 Select an action [Analyze]:
  [0] Analyze
  [1] Full Analyze
  [2] Fetch - Fetch missing remote avatars
  [3] Fix - Fix remote accounts without avatar record
  [4] Sync - Store latest remote avatars
 > 4

Selected: Sync - Store latest remote avatars

@foozmeat
Copy link

foozmeat commented Mar 8, 2023

I'm no longer able to repro this issue

@dansup dansup closed this as completed in 28bf864 Apr 4, 2023
yourivdlans pushed a commit to yourivdlans/pixelfed that referenced this issue Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants