New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uncaught BadRequestException even though upload works #42

Closed
lbearl opened this Issue Apr 3, 2017 · 25 comments

Comments

Projects
None yet
3 participants
@lbearl

lbearl commented Apr 3, 2017

I'm getting the following error every time I attempt to upload a file:

PHP Fatal error: Uncaught exception 'Google\\Cloud\\Core\\Exception\\BadRequestException' with message 'Client error:PUT https://www.googleapis.com/upload/storage/v1/b/xxx/o?uploadType=resumable&upload_id=AEnB2Uxxxrs2ddVLKbe_jBSlOHJHudM0DGN-Ww5_4JOMbcEWjgT1g_2D226F7B1k349bjp6PZtyHFmXepDUG-YnsRoHkKgKQoPZVw` resulted in a 400 Bad Request response:\n<title>Error 400 (Bad Request)!!1</title>\n' in /var/www/lmydir/wp-content/plugins/gcs/vendor/google/cloud/src/Core/RequestWrapper.php:232\nStack trace:\n#0 /var/www/lmydir/wp-content/plugins/gcs/vendor/google/cloud/src/Core/RequestWrapper.php(145): Google\Cloud\Core\RequestWrapper->convertToGoogleException(Object(GuzzleHttp\Exception\ClientException))\n#1 /var/www/mydir/wp-content/plugins/gcs/vendor/google/cloud/src/Core/Upload/StreamableUploader.php(71): Google\Cloud\Core\RequestWrapper->send(Object(GuzzleHttp\Psr7\Request), Array)\n#2 /var/www/mydir/wp-content/plugins/gcs/vendor/google/cloud/src/Storage/WriteStream.php(66): Google\Cloud\Core\ in /var/www/mydir/wp-content/plugins/gcs/vendor/google/cloud/src/Core/RequestWrapper.php on line 232, referer: https://mydir/wp-admin/post-new.php`

Note that the upload to GCP Storage is actually successful, but due to the error Wordpress doesn't record the file as having been uploaded. I am hosting wordpress using a GCP Compute Engine VM.

Please let me know if there are any other details I can provide.

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 11, 2017

Member

@lbearl

Do your Compute Engine service account have the write access to the GCE bucket in question?

Note that the upload to GCP Storage is actually successful

Do you really see the files in the GCS bucket?

I can not reproduce your problem so far. Does it happen for every image file you try to upload? If it happens only for certain images, can you send me the image file which fails? Can you share the actual operations you did when it happened? Even, is it possible to invite me (tmatsuo@google.com) to your cloud project so that I can investigate?

Member

tmatsuo commented Apr 11, 2017

@lbearl

Do your Compute Engine service account have the write access to the GCE bucket in question?

Note that the upload to GCP Storage is actually successful

Do you really see the files in the GCS bucket?

I can not reproduce your problem so far. Does it happen for every image file you try to upload? If it happens only for certain images, can you send me the image file which fails? Can you share the actual operations you did when it happened? Even, is it possible to invite me (tmatsuo@google.com) to your cloud project so that I can investigate?

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

@tmatsuo

Thanks for the reply. I double checked that the service account that the VM is running under has write access to that bucket. I also used gsutil to upload a test file to that bucket.

I can reliably reproduce the issue with the following steps:

  1. Go to "Media > Add New" (in wordpress admin)
  2. Select any file the file will appear to upload and then a message will appear saying "HTTP Error"
  3. In Google Storage, the image will appear (although I just noticed that it will be 0 bytes in size)

I've added you as a project viewer to my account. Let me know if you need access to the WP admin dashboard as well.

Thanks for your help!

lbearl commented Apr 12, 2017

@tmatsuo

Thanks for the reply. I double checked that the service account that the VM is running under has write access to that bucket. I also used gsutil to upload a test file to that bucket.

I can reliably reproduce the issue with the following steps:

  1. Go to "Media > Add New" (in wordpress admin)
  2. Select any file the file will appear to upload and then a message will appear saying "HTTP Error"
  3. In Google Storage, the image will appear (although I just noticed that it will be 0 bytes in size)

I've added you as a project viewer to my account. Let me know if you need access to the WP admin dashboard as well.

Thanks for your help!

@hyperpress

This comment has been minimized.

Show comment
Hide comment
@hyperpress

hyperpress Apr 12, 2017

hyperpress commented Apr 12, 2017

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

@hyperpress thanks for the help. I tried everything you suggested, but unfortunately that didn't seem to do it. As far as the developer tools go, they just report a 500.

@tmatsuo one thing I noticed while going through the logs is that the upload request id is the same for every upload, is that correct?

Client error: 'PUT https://www.googleapis.com/upload/storage/v1/b/lukebearl/o?uploadType=resumable&upload_id=AEnB2Uqz40k68UJbs<snipsnip>47rj9ZuYG5ZVDp9tTd805EZHust1zECoWmY' resulted in a '400 Bad Request'

lbearl commented Apr 12, 2017

@hyperpress thanks for the help. I tried everything you suggested, but unfortunately that didn't seem to do it. As far as the developer tools go, they just report a 500.

@tmatsuo one thing I noticed while going through the logs is that the upload request id is the same for every upload, is that correct?

Client error: 'PUT https://www.googleapis.com/upload/storage/v1/b/lukebearl/o?uploadType=resumable&upload_id=AEnB2Uqz40k68UJbs<snipsnip>47rj9ZuYG5ZVDp9tTd805EZHust1zECoWmY' resulted in a '400 Bad Request'

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

@lbearl Do you use the service account key for wordpress on the VM? Do you set environment variable GOOGLE_APPLICATION_CREDENTIALS?

Member

tmatsuo commented Apr 12, 2017

@lbearl Do you use the service account key for wordpress on the VM? Do you set environment variable GOOGLE_APPLICATION_CREDENTIALS?

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

@tmatsuo I'm not using that service accout. I just removed it. Since the VM is running inside of GCP I believe the Cloud API access scopes for VM are sufficient to grant access to both SQL and Storage?

lbearl commented Apr 12, 2017

@tmatsuo I'm not using that service accout. I just removed it. Since the VM is running inside of GCP I believe the Cloud API access scopes for VM are sufficient to grant access to both SQL and Storage?

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

@lbearl Still I'm not sure how to reproduce your issue. I access the VM, but it says "Error establishing a database connection" or something like that.

Member

tmatsuo commented Apr 12, 2017

@lbearl Still I'm not sure how to reproduce your issue. I access the VM, but it says "Error establishing a database connection" or something like that.

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

@tmatsuo yeah... turns out I had used that service account when I configured the sql cloud proxy. I removed the proxy and white listed the web servers external IP on the SQL access control. Sorry about that. That part should be working again.

lbearl commented Apr 12, 2017

@tmatsuo yeah... turns out I had used that service account when I configured the sql cloud proxy. I removed the proxy and white listed the web servers external IP on the SQL access control. Sorry about that. That part should be working again.

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

@lbearl
No problem. Is it possible for you to check the environment variables? Just make sure there's no GOOGLE_APPLICATOIN_CREDENTIALS envvar.

Member

tmatsuo commented Apr 12, 2017

@lbearl
No problem. Is it possible for you to check the environment variables? Just make sure there's no GOOGLE_APPLICATOIN_CREDENTIALS envvar.

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

@tmatsuo yeah, I executed echo $GOOGLE_APPLICATION_CREDENTIALS and it returned nothing, so that should mean it isn't set.

lbearl commented Apr 12, 2017

@tmatsuo yeah, I executed echo $GOOGLE_APPLICATION_CREDENTIALS and it returned nothing, so that should mean it isn't set.

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

@lbearl

The shell env might different from the webserver's env. Is it possible to put a php file with phpinfo() and access that file via the web server and check the env vars there?

Member

tmatsuo commented Apr 12, 2017

@lbearl

The shell env might different from the webserver's env. Is it possible to put a php file with phpinfo() and access that file via the web server and check the env vars there?

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

@tmatsuo I created a file with phpinfo() and ctrl-f for "GOOGLE" returns nothing.

lbearl commented Apr 12, 2017

@tmatsuo I created a file with phpinfo() and ctrl-f for "GOOGLE" returns nothing.

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

@lbearl Thanks, but unfortunately I don't know why. Is it possible for me to ssh into the VM so that I can try debugging?

Member

tmatsuo commented Apr 12, 2017

@lbearl Thanks, but unfortunately I don't know why. Is it possible for me to ssh into the VM so that I can try debugging?

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

@tmatsuo yeah, go for it. Do you need me to grant you additional privileges?

lbearl commented Apr 12, 2017

@tmatsuo yeah, go for it. Do you need me to grant you additional privileges?

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

I think the easiest way is to give me an editor access, or add this ssh key to the metadata:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC91n96GIrwGSYmdQBtVM/IUAz0GQiBUxtlmoM6qoJNBRADpT18XvOxdChf7AlIv4eRxzgCr0BuKeWa1AV/8s8cdLPVZVqUVPibz6jK6HKtlkNfTFKZls8wV5OqlOccgStGRftnF2VvshDu0g9S3Xp3OIBqx66dlzNBEjFyp6jLpDhTdIetvRFgM5l8DAr8Frr98oR66ixrUet65zMvy7y3LJpmsCAUMSP+VYbBVYGFc1dpfTrk7yWWEc3mEv6Nef/Jo9we8ABlRa1VELlaegGsG2SUY32C6cpX7VtRJJCvTRQ1Cz+irBje8d+ekrdyJdsmZrh3jbfTR/iIWpbK7BWA1gL9rXGEO/suz3Ay+aV4NFGl6PApG8Lq/0o2FCY9j0SPqSPs1RZg7J5iDJI74uP86NwmGfi0PyBMJ2YRatsfZpVPDjxzqaU/Y+9nQoyUk03B5DgL11i9Vk/c1Q+wK5SAxHO7g6yDvxEg6rXW+cPYL1J+nbAQOFUSradgMlpEASpcuDowgkpy14SzIuwwfP1F3hna5HJnmtS2JA0Fw+LWyUOGpglvvShZ4nugWDsIqa0K0e4h5DuXrrLLtAXnTg5lTzhWoIqiFXBM4Z1BAd8CsGaFsL59X0ScAB81iy7fk7YtNpAgu0c+KYhFthXGy1c4ojxw5Lwd1bzzo3HPXn4rEQ== tmatsuo@t-glaptop
Member

tmatsuo commented Apr 12, 2017

I think the easiest way is to give me an editor access, or add this ssh key to the metadata:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC91n96GIrwGSYmdQBtVM/IUAz0GQiBUxtlmoM6qoJNBRADpT18XvOxdChf7AlIv4eRxzgCr0BuKeWa1AV/8s8cdLPVZVqUVPibz6jK6HKtlkNfTFKZls8wV5OqlOccgStGRftnF2VvshDu0g9S3Xp3OIBqx66dlzNBEjFyp6jLpDhTdIetvRFgM5l8DAr8Frr98oR66ixrUet65zMvy7y3LJpmsCAUMSP+VYbBVYGFc1dpfTrk7yWWEc3mEv6Nef/Jo9we8ABlRa1VELlaegGsG2SUY32C6cpX7VtRJJCvTRQ1Cz+irBje8d+ekrdyJdsmZrh3jbfTR/iIWpbK7BWA1gL9rXGEO/suz3Ay+aV4NFGl6PApG8Lq/0o2FCY9j0SPqSPs1RZg7J5iDJI74uP86NwmGfi0PyBMJ2YRatsfZpVPDjxzqaU/Y+9nQoyUk03B5DgL11i9Vk/c1Q+wK5SAxHO7g6yDvxEg6rXW+cPYL1J+nbAQOFUSradgMlpEASpcuDowgkpy14SzIuwwfP1F3hna5HJnmtS2JA0Fw+LWyUOGpglvvShZ4nugWDsIqa0K0e4h5DuXrrLLtAXnTg5lTzhWoIqiFXBM4Z1BAd8CsGaFsL59X0ScAB81iy7fk7YtNpAgu0c+KYhFthXGy1c4ojxw5Lwd1bzzo3HPXn4rEQ== tmatsuo@t-glaptop
@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

I've promoted you to have project editor permissions.

lbearl commented Apr 12, 2017

I've promoted you to have project editor permissions.

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

Thanks. I think I started understand what's happening. My hypothesis now is it may need full storage permission for the resumable upload.

Member

tmatsuo commented Apr 12, 2017

Thanks. I think I started understand what's happening. My hypothesis now is it may need full storage permission for the resumable upload.

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 12, 2017

That sounds plausible. I'll change the VM's API access scopes and report back. Thanks for your help tonight.

EDIT: I tried but after the VM came back up I got the same exception. The permissions should have full access to storage:
image

lbearl commented Apr 12, 2017

That sounds plausible. I'll change the VM's API access scopes and report back. Thanks for your help tonight.

EDIT: I tried but after the VM came back up I got the same exception. The permissions should have full access to storage:
image

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

Good news is that I reproduced the same issue with my own compute instance :)

Bad news is that I still don't know why. My instance has Full cloud access, but it still fails. I assume it's not a permission related.

https://cloud.google.com/storage/docs/best-practices

If you use Google Compute Engine instances with processes that POST to Cloud Storage to initiate a resumable upload, then you should use Compute Engine instances in the same locations as your Cloud Storage buckets. You can then use a geo IP service to pick the Compute Engine region to which you route customer requests, which will help keep traffic localized to a geo-region.

Maybe it's related, but we're all using multi regional storage bucket in US and GCE instance in us, so I'm at a loss. I'll investigate further tomorrow.

Member

tmatsuo commented Apr 12, 2017

Good news is that I reproduced the same issue with my own compute instance :)

Bad news is that I still don't know why. My instance has Full cloud access, but it still fails. I assume it's not a permission related.

https://cloud.google.com/storage/docs/best-practices

If you use Google Compute Engine instances with processes that POST to Cloud Storage to initiate a resumable upload, then you should use Compute Engine instances in the same locations as your Cloud Storage buckets. You can then use a geo IP service to pick the Compute Engine region to which you route customer requests, which will help keep traffic localized to a geo-region.

Maybe it's related, but we're all using multi regional storage bucket in US and GCE instance in us, so I'm at a loss. I'll investigate further tomorrow.

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

Filed a bug on the google-cloud-php repo: googleapis/google-cloud-php#447

Member

tmatsuo commented Apr 12, 2017

Filed a bug on the google-cloud-php repo: googleapis/google-cloud-php#447

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

@lbearl

Hi, I think we found a workaround. Can you enable curl extension and try it again? If you're just using debian package, you can just do sudo apt-get update && sudo apt-get install -y php5-curl.

Member

tmatsuo commented Apr 12, 2017

@lbearl

Hi, I think we found a workaround. Can you enable curl extension and try it again? If you're just using debian package, you can just do sudo apt-get update && sudo apt-get install -y php5-curl.

@hyperpress

This comment has been minimized.

Show comment
Hide comment
@hyperpress

hyperpress Apr 12, 2017

hyperpress commented Apr 12, 2017

@tmatsuo

This comment has been minimized.

Show comment
Hide comment
@tmatsuo

tmatsuo Apr 12, 2017

Member

@hyperpress Thanks for the additional data point!

Ideally the underlining library should just work without curl, so we'll keep investigating why, but in general installing curl is easy and most of the users won't have a problem doing so.

Member

tmatsuo commented Apr 12, 2017

@hyperpress Thanks for the additional data point!

Ideally the underlining library should just work without curl, so we'll keep investigating why, but in general installing curl is easy and most of the users won't have a problem doing so.

@hyperpress

This comment has been minimized.

Show comment
Hide comment
@hyperpress

hyperpress Apr 12, 2017

hyperpress commented Apr 12, 2017

@lbearl

This comment has been minimized.

Show comment
Hide comment
@lbearl

lbearl Apr 13, 2017

@tmatsuo thanks for all your help! I've installed curl and restarted apache and now it is working. @hyperpress thanks for all your help as well.

lbearl commented Apr 13, 2017

@tmatsuo thanks for all your help! I've installed curl and restarted apache and now it is working. @hyperpress thanks for all your help as well.

@lbearl lbearl closed this Apr 13, 2017

lbearl added a commit to lbearl/wordpress-plugins that referenced this issue Apr 13, 2017

Including installation instructions
Updated readme to include instructions for resolving #42.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment