Gatsby 4: wordpress source looses auth credientials during "validating engines" phase

[joernroeder]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

I'm in the process of moving a multi source page from gatsby 3 to v4 and I am currently running into issues with the wordpress source. The following is my current process of digging through it. I've found the root of the (first) issue eventually and will try to lay out my findings in a linear way, even tho I did not find them in this order while I was digging through the source:

The WP Setup

I have a WP instance secured behind basic auth. I'm using authentication as described here without any issue for a while.
https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-wordpress/docs/plugin-options.md#authhtaccess

{
  resolve: `gatsby-source-wordpress`,
  options: {
    auth: {
      htaccess: {
        username: `admin`,
        password: `1234strong_password`,
      },
    },
  },
}

The Build Error

since switching to v4 (currently gatsby 4.4, wp-source v6.4.0) my builds are failing with

error  gatsby-source-wordpress  Request failed with status code 401.

I always thought this is due some incompatibility with the source plugin or a rate limiting on the wp hosting side due to more processes being involved now and did not pay any attention, hoped it would resolve itself with the next gatsby/wp-source version — but it did not.

I finally looked more closely at this over the holidays, logged out the htaccessCredentials.

gatsby/packages/gatsby-source-wordpress/src/utils/fetch-graphql.ts

Line 389 in 4a765b5

const htaccessCredentials = pluginOptions.auth.htaccess

During the sourcing phase of gatsby this looked fine,

htaccessCredentials { username: 'myuser', password: 'mypw' }
success  gatsby-source-wordpress  diff schemas - 0.983s

but it failed at the very end during rendering engine validation 🤔

⠙ Validating Rendering Engines

my console.log --> htaccessCredentials { username: null, password: null }

error  gatsby-source-wordpress  Request failed with status code 401.

 Your WordPress instance may be protected with HTTP Basic authentication.
 If it is you will need to add the following to your plugin options:

        {
          resolve: `gatsby-source-wordpress`,
          options: {
            auth: {
              htaccess: {
                username: process.env.HTTPBASICAUTH_USERNAME,
                password: process.env.HTTPBASICAUTH_PASSWORD,
              }
            }
          }
        }

GraphQL request to https://mydomain.com/graphql failed.

Please ensure the following statements are true
  - your WordPress URL is correct in gatsby-config.js
  - your server is responding to requests
  - WPGraphQL and WPGatsby are installed and active in your WordPress backend
  - Your WordPress debug.log does not contain critical errors
not finished  gatsby-source-wordpress  ingest WPGraphQL schema - 0.574s
failed Validating Rendering Engines - 9.991s

 ERROR #98001  WEBPACK

Built Rendering Engines failed validation.

So why are they not there anymore? Plugin options are passed in, and are coming from

gatsby/packages/gatsby-source-wordpress/src/utils/fetch-graphql.ts

Line 694 in 4a765b5

const { helpers, pluginOptions } = store.getState().gatsbyApi

So let's take a look of the store:

{
  url: 'https://mydomain.com/graphql',
  verbose: true,
  debug: {
    throwRefetchErrors: false,
    graphql: {
      showQueryOnError: false,
      showQueryVarsOnError: false,
      copyQueryOnError: false,
      panicOnError: false,
      onlyReportCriticalErrors: true,
      copyNodeSourcingQueryAndExit: false,
      writeQueriesToDisk: false,
      copyHtmlResponseOnError: false,
      printIntrospectionDiff: false
    },
    timeBuildSteps: false,
    disableCompatibilityCheck: false,
    preview: false
  },
  develop: {
    nodeUpdateInterval: 5000,
    hardCacheMediaFiles: true,
    hardCacheData: false
  },
  production: {
    hardCacheMediaFiles: false,
    allow404Images: false,
    allow401Images: false
  },
  auth: { htaccess: { username: null, password: null } },
  schema: {
    queryDepth: 15,
    circularQueryLimit: 5,
    typePrefix: 'Wp',
    timeout: 90000,
    perPage: 100,
    requestConcurrency: 5,
    previewRequestConcurrency: 2
  },

store is there, also custom limits, concurrency, url etc but the auth data are missing. 🤔
After realizing that the store already lost the auth information I continued my digging and found:

gatsby/packages/gatsby-source-wordpress/src/steps/process-and-validate-plugin-options.ts

Line 171 in 492068b

delete pluginOptions.auth

commenting out that line let me proceed that step

success Validating Rendering Engines - 18.540s
success Caching Webpack compilations - 0.001s
success run queries in workers - 17.314s - 108/108 6.24/s
success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 44.514s - 312/312 7.01/s

But my build fails at another phase:

  Error: Assertion failed: all worker queries are not dirty (worker #8)

  - queries.ts:391 assertCorrectWorkerState
    [project]/[gatsby]/src/redux/reducers/queries.ts:391:13

  - queries.ts:224 queriesReducer
    [project]/[gatsby]/src/redux/reducers/queries.ts:224:7

  - redux.js:536 combination
    [project]/[redux]/lib/redux.js:536:29

  - redux.js:296 next
    [project]/[redux]/lib/redux.js:296:22

  - index.ts:72
    [project]/[gatsby]/src/redux/index.ts:72:68

  - index.js:27 Object.dispatch
    [project]/[redux-thunk]/lib/index.js:27:16

  - pool.ts:117 mergeWorkerState
    [project]/[gatsby]/src/utils/worker/pool.ts:117:13

  - runMicrotasks

  - task_queues:96 processTicksAndRejections
    node:internal/process/task_queues:96:5

  - build.ts:296 build
    [project]/[gatsby]/src/commands/build.ts:296:5


not finished Merge worker state - 0.048s

I'll continue to dig into this, but as it's time consuming i am not sure when and for how long. I'd love to take advantage of the features v4 brings, so there is some need on my side to get this resolved.

I don't have a full understanding of the gatsby architecture but why this this being removed at all? or when will this be shared with the browser?

// remove auth from pluginOptions so we don't leak into the browser
delete pluginOptions.auth

Reproduction Link

todo

Steps to Reproduce

1. use a WP source protected with Basic Auth

...

Expected Result

build to succeed

Actual Result

build fails

Environment

System:
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.13.0
    Yarn: 1.18.0
    npm: 8.1.0 - ~/.nvm/versions/node/v16.13.0/bin/npm

Config Flags

No response

[joernroeder]

Hey @wardpeet, sorry for pinging your directly but as you wrote the most critical line in this issue you might be able to help me out here.

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[joernroeder]

not stale. still an issue…

[LekoArts]

I'll let the team know 👍

[TylerBarnes]

@joernroeder I'm not able to reproduce this. Can you share a minimal reproduction as well as the output of gatsby info?

[TylerBarnes]

you can use the info of this test site to make the repro

url: https://gatsbypreview.wpengine.com/graphql
username: gatsbypreview
password: testpass

It's just a test site and the user/pass is only for the server basic auth so it's fine to share it here.

[joernroeder]

@TylerBarnes Thanks for lloking into this and providing the wp instance. I'll look into it this upcoming week and will come back to you.

[joernroeder]

Hey @TylerBarnes, you're right this issue seem to have disappeared with the latest version — gatsby ^4.7.2 with gatsby-source-wordpress ^6.7.0. Again, thanks for looking into this!

Strangely, the moment I swap out the wp creds and url with my url I run into issues, unrelated to the htaccess problem. It seems those occur while fetching media items and the subsequent run of the gatsby-plugin-sharp.IMAGE_PROCESSING jobs and are probably related to the following and don't appear with your wp project as it does not contain any media items.

#34215
#34610
#34051

I'll leave this issue open this week until I was able to do some more testing. Here is the projects gatsby-4 branch in the cloud If that's helpful https://www.gatsbyjs.com/dashboard/d51a9a01-a8f2-4842-92b6-da5b9f8ac9da/sites/3bc1e35a-011b-4bdf-b904-c4da011f221c/builds/67f518b8-8c09-40f7-b5f5-9d16e10622cb/details#all

[joernroeder]

Do you think it makes sense to add a couple of images to your wp instance so we can test the same behavior there as well?

[joernroeder]

Hey @TylerBarnes, I'm still experimenting with gatsby 4 and try to get the site ported over. I'm on gatsby-source-wordpress@6.10.2 now and seeing the error in a slightly different flavor with the IMAGE_CDN job now which seems to loose the htaccess creds.

https://www.gatsbyjs.com/dashboard/d51a9a01-a8f2-4842-92b6-da5b9f8ac9da/sites/3bc1e35a-011b-4bdf-b904-c4da011f221c/builds/45e63e92-5ec9-450c-9834-c878bbcc6aba/details

[joernroeder]

@TylerBarnes can you add some images to the wp test instance mentioned above and I'll spin up a dedicated instance with it on the cloud to see if that also breaks. currently your wp instance does not contain any media items.

[TylerBarnes]

Closing as a duplicate of #35151. I did add a featured image on a post in the test site though!