[gatsby-source-drupal] file download not working if using baseUrl of a proxy with path

[Vacilando]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

Situation: working Drupal 9 sites, latest stable gatsby-source-drupal. Gatsby slowing down Drupal during build. Need for an API Gateway.

Set up Amazon API Gateway, tested the connection, the proxy URL has format like https://xyz.execute-api.eu-west-1.amazonaws.com/prod
There is no way to omit the "/prod" path in the API Gateway, as it signifies the environment.

Build works locally and on Gatsby Cloud as long as the Drupal site does not have any images/files.

When building using content from Drupal sites with images/files, the build fails (locally and on Gatsby Cloud) with error like
[gatsby-source-drupal] Encountered an unrecoverable error while generating Gatsby Image CDN fields for url https://xyz.execute-api.eu-west-1.amazonaws.com/sites/default/files/media-icons/generic/video.png. See above for additional information.

Note that the URL does NOT have "/prod" in there. That's the problem.
When I manually try with
https://xyz.execute-api.eu-west-1.amazonaws.com/prod/sites/default/files/media-icons/generic/video.png
then it does work.

So the problem is that

• sometimes we have to use baseUrl with a path
• it works fine for JSON API
• it fails when downloading files from Drupal

Reproduction Link

https://github.com/repo_is_not_public

Steps to Reproduce

1. Set up a Drupal 9 site with some images
2. Set up a Gatsby site pulling data from Drupal using plugin gatsby-source-drupal
3. Set up a proxy (e.g. using AWS API Gateway) with a path
4. gatsby clean && gatsby develop

Expected Result

File URLs like:
https://xyz.execute-api.eu-west-1.amazonaws.com/prod/sites/default/files/media-icons/generic/video.png

Actual Result

File URLs like:
https://xyz.execute-api.eu-west-1.amazonaws.com/sites/default/files/media-icons/generic/video.png

Environment

System:
    OS: Linux 5.10 Ubuntu 20.04.3 LTS (Focal Fossa)
    CPU: (8) x64 Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 14.20.0 - ~/.nvm/versions/node/v14.20.0/bin/node
    npm: 8.19.2 - ~/.nvm/versions/node/v14.20.0/bin/npm
  npmPackages:
    gatsby: ^4.24.1 => 4.24.1
    gatsby-plugin-canonical-urls: ^4.24.0 => 4.24.0
    gatsby-plugin-gatsby-cloud: ^4.24.0 => 4.24.0
    gatsby-plugin-google-analytics: ^4.24.0 => 4.24.0
    gatsby-plugin-image: ^2.24.0 => 2.24.0
    gatsby-plugin-security-txt: ^1.1.8 => 1.1.8
    gatsby-plugin-sitemap: ^5.24.0 => 5.24.0
    gatsby-source-drupal: ^5.25.0 => 5.25.0
    gatsby-source-filesystem: ^4.24.0 => 4.24.0
    gatsby-wikipedia-fetcher: ^1.4.0 => 1.4.0
  npmGlobalPackages:
    gatsby-cli: 4.24.0

Config Flags

PRESERVE_FILE_DOWNLOAD_CACHE
FAST_DEV
PARALLEL_SOURCING
DEV_SSR

[Vacilando]

I think for this to work we will need a little feature added to gatsby-source-drupal -- see #36811

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[Vacilando]

Will be fixed by #36819

[Vacilando]

This has just been released in plugin gatsby-source-drupal (for Gatsby 5 as well as backported to Gatsby 4).
In the config we can now provide option proxyUrl containing a proxy (CDN, API Gateway, etc.) URL to cache the Drupal API responses.

In our use case (a Drupal 9 site with about 100k nodes) a fresh fetch from a respectable Drupal server (32 GB RAM, 8 vCPUs) took about 170 seconds (build phase "Fetch all data from Drupal").
Using a CDN (CloudFront) we've shortened the fetch phase to 17s (10x faster!)