gatsby-source-wordpress: Build failure - WP URL might accidentally be stripped of subdomain

[etnichols]

Hi there,

I'm seeing the following build failure using v.3.3.0 of the plugin (also repro'ed using v.3.0.0 version).

Note: I use this plugin with a wordpress instance hosted at wordpress.leadtoreadkc.org, while the actual Gatsby site is hosted at leadtoreadkc.org. I suspect this fact might be causing the error, given the error message below

evannichols:_dev/lead-to-read ‹master*›$ gatsby develop
success open and validate gatsby-configs - 0.064s
warn Warning: there are unknown plugin options for "gatsby-source-wordpress-experimental":
debug.graphql.copyHtmlResponseOnError
success load plugins - 1.423s
success onPreInit - 0.046s
success initialize cache - 0.091s
success copy gatsby files - 0.120s
success onPreBootstrap - 0.028s
success  gatsby-source-wordpress  ensuring plugin requiremenst are met - 5.073s
⠀
info  gatsby-source-wordpress

	This is either your first build or the cache was cleared.
	Please wait while your WordPress data is synced to your Gatsby cache.

	Maybe now's a good time to get up and stretch? :D

success  gatsby-source-wordpress  ingest WPGraphQL schema - 2.438s
success createSchemaCustomization - 7.570s
success  gatsby-source-wordpress  fetch root fields - 2.494s
success  gatsby-source-wordpress  ContentType - 3.299s - fetched 3
success  gatsby-source-wordpress  Comment - 6.783s - fetched 16
success  gatsby-source-wordpress  Category - 7.428s - fetched 13
success  gatsby-source-wordpress  Menu - 1.806s - fetched 1
success  gatsby-source-wordpress  Page - 14.492s - fetched 29
success  gatsby-source-wordpress  PostFormat - 2.650s - fetched 0
success  gatsby-source-wordpress  Tag - 10.174s - fetched 55
success  gatsby-source-wordpress  Post - 12.994s - fetched 59
success  gatsby-source-wordpress  Taxonomy - 2.348s - fetched 3
success  gatsby-source-wordpress  UserRole - 2.402s - fetched 0
success  gatsby-source-wordpress  User - 3.562s - fetched 4

 ERROR

HTTPError: Response code 404 (Not Found)
    at Request._onResponseBase (/Users/evannichols/_dev/lead-to-read/node_modules/gatsby-source-wordpress-experimental/node_modules/got/dist/source/core/index.js:896:31)
    at Request._onResponse (/Users/evannichols/_dev/lead-to-read/node_modules/gatsby-source-wordpress-experimental/node_modules/got/dist/source/core/index.js:931:24)
    at ClientRequest.<anonymous> (/Users/evannichols/_dev/lead-to-read/node_modules/gatsby-source-wordpress-experimental/node_modules/got/dist/source/core/index.js:945:23)
    at Object.onceWrapper (events.js:299:26)
    at ClientRequest.emit (events.js:214:15)
    at ClientRequest.origin.emit (/Users/evannichols/_dev/lead-to-read/node_modules/gatsby-source-wordpress-experimental/node_modules/@szmarczak/http-timer/dist/source/index.js:39:20)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:583:27)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:115:17)
    at TLSSocket.socketOnData (_http_client.js:456:22)
    at TLSSocket.emit (events.js:209:13)
    at addChunk (_stream_readable.js:305:12)
    at readableAddChunk (_stream_readable.js:286:11)
    at TLSSocket.Readable.push (_stream_readable.js:220:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:182:23) {
  name: 'HTTPError',
  code: undefined,
  timings: {
    start: 1606519178323,
    socket: 1606519178323,
    lookup: 1606519178334,
    connect: 1606519178340,
    secureConnect: 1606519178352,
    upload: 1606519178353,
    response: 1606519178370,
    end: 1606519178383,
    error: undefined,
    abort: undefined,
    phases: {
      wait: 0,
      dns: 11,
      tcp: 6,
      tls: 12,
      request: 1,
      firstByte: 17,
      download: 13,
      total: 60
    }
  }
}

// (from evan) - IMPORTANT! Why is it trying to download image from leadtoreadkc.org
// instead of wordpress.leadtoreadkc.org?
warn Received the below error when trying to fetch
https://www.leadtoreadkc.org/wp-content/uploads/2019/10/roofotp-10-1024x683.jpg
from Post #2709 "Readers on the Rooftop Happy Hour"

https://wordpress.leadtoreadkc.org/wp-admin/post.php?post=2709&action=edit



 ERROR

 gatsby-source-wordpress  undefined

not finished source and transform nodes - 96.990s
not finished  gatsby-source-wordpress  fetching nodes - 96.847s - 239 total
not finished  gatsby-source-wordpress  creating nodes - 58.324s
not finished  gatsby-source-wordpress  MediaItem - 58.333s - fetched 56
not finished Downloading remote files - 55.274s
not finished Generating image thumbnails - 54.957s

• It's failing with a NOT_FOUND when trying to fetch https://www.leadtoreadkc.org/wp-content/uploads/2019/10/roofotp-10-1024x683.jpg, (it's missing the subdomain).
• https://wordpress.leadtoreadkc.org/wp-content/uploads/2019/10/roofotp-10-1024x683.jpg does indeed exist, but trying to visit https://www.wordpress.leadtoreadkc.org/wp-content/uploads/2019/10/roofotp-10-1024x683.jpg in the browser hangs... I think due to the www prefix.

Is the plugin stripping out the subdomain for some reason? Or is this a configuration issue on my part?

Please let me know if I can provide any more debugging info.

I have shared a branch of the affected repo that reproduces this issue with you (https://github.com/etnichols/lead-to-read/tree/wp-exp-bug).

[etnichols]

Additional notes:

• I was upgrading gatsby-source-wordpress-experimental from 1.3.6 to 3.3.0
• Updated WP Gatsby plugin to Version 0.6.7 (I don't remember what previous version was)
• Updated WPGraphQL plugin to Version 1.0 (I don't remember what previous version was)
• Verified that it wasn't an issue with that image in particular -- I deleted it from the post it was included in, tried again, and observed the same error but with https://www.leadtoreadkc.org/wp-content/uploads/2019/10/rooftop-23-683x1024.jpg.

[etnichols]

Doing a little more digging:

Looks like the error gets thrown here, in fetchNodeHtmlImageMediaItemNodes.

The URL to fetch is htmlImgSrc which is produced via ensureSrcHasHostname.

It's using url.parse to extract the protocol and host from the wpUrl... nothing unexpected here, and confirmed that url.parse should behave correctly given my WP instance:

> url.parse('https://wordpress.leadtoreadkc.org')
Url {
  protocol: 'https:',
  slashes: true,
  auth: null,
  host: 'wordpress.leadtoreadkc.org',
  port: null,
  hostname: 'wordpress.leadtoreadkc.org',
  hash: null,
  search: null,
  query: null,
  pathname: '/',
  path: '/',
  href: 'https://wordpress.leadtoreadkc.org/'
}

So this leads me to believe that the subdomain in my WP url -- the wordpress in https://wordpress.leadtoreadkc.org -- is getting erroneously replaced with www somewhere up the line.

wpUrl is a member variable of remoteSchema. I could use some help here... I'm not certain how the remoteSchema gets populated. This might lead us closer to the error.

[etnichols]

(warning: incredibly hacky workaround)

I edited this section of process-node.js to check for the incorrect www subdomain and replace it with wordpress:

for (const {
    cheerioImg,
    match
  } of cheerioImages) {
    let htmlImgSrc = ensureSrcHasHostname({
      src: cheerioImg.attribs.src,
      wpUrl
    });

    if(htmlImgSrc.includes('www')){
      helpers.reporter.warn(`\nHTML img src includes www subdomain (incorrect): ${htmlImgSrc}. Fixing...`);
      htmlImgSrc = htmlImgSrc.replace('www', 'wordpress')
    }

    let imageNode = pickNodeBySourceUrlOrCheerioImg({
      url: htmlImgSrc,
      cheerioImg,
      mediaItemNodes
    });

    if (!imageNode && htmlImgSrc) {
      helpers.reporter.warn(`\nWe didn't get a media item node for this image, we need to fetch it and create a file node for it with no media item node. htmlImgSrc for node: ${htmlImgSrc}`)
      // if we didn't get a media item node for this image,
      // we need to fetch it and create a file node for it with no
      // media item node.
      try {
       // tries to fetch the img resource and now it succeeds
     }
...
}

But this is incredibly brittle 😄 the longer term solution is fixing instantiation of wpUrl.

[TylerBarnes]

Thanks for opening this issue and including a reproduction GraphQL url! We're currently focused on a few issues but will be fixing this before our stable release (the date of which is TBD but coming up soon)

[etnichols]

Thanks for the reply and sounds good to me!

[etnichols]

Friendly bump -- are there any updates regarding this bug or expected date of the stable release?

[etnichols]

Any updates on this? Was this potentially fixed in the stable gatsby-source-wordpress@4.0.0 release?

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜