Allow Gatsby Source Kontent Plugin to filter by collection

[anthonyshughes]

Motivation

We use one Kontent project with multiple collections for mutiple sites. We have a separate build process for each site. This means that each site pulls all the data down for all the sites even though it is not needed. We can filter by language but all our sites use the en-gb language as well as their own native language so all en-gb content is pulled even if it is not in a relevant collection

Proposed solution

Enable filtering on collection as well as language

[bwlng]

Curious if adding support for query params would be a good way to approach this. On our sites we would benefit from filtering content items by workflow steps (such as archive — https://preview-deliver.kontent.ai/{{KENTICO_PROJECT_ID}}/items-feed?language=en&system.workflow_step=archived) to limit the content sourced for builds. Collections can also be filtered this way according to the API. A more flexible solution that relies on the API's robust filtering method could enable a variety of capabilities.

One wrinkle to this approach is that it appears filtering by language is handled with a query param, so perhaps it would need to be documented as an optional additionalFilterParams or similar to avoid complicating the implementation.

[Simply007]

Hello @anthonyshughes and @bwlng,

thank you for reaching out!

I was thinking about the robustness of the solution. And @bwlng's idea about extending the configuration is pretty easy to implement but the robustness would not be really great - I will try to come up with something more complex, like a hook, but we can fall back to the easy string-based configuration for the queries. - Can you think of any other filter you might want to use? IMO it makes sense for filtering based on system properties, right?

But in general, I have identified some possible problematic areas I would like to discuss prior to the implementation.

Cross collection Links

• How about linking an item from a different collection - Collection1 -> Collection2 using linked items element or rich text element. This item would not be in the model is that OK?

Cross Links that aims to filtered-out item

• How about linking an item from another workflow (any s property) using linked items element or rich text element. This item that does not fit the filter would not be in the model is that OK?

Webhook handling

• Currently there is no way of configuring webhooks to be fired only for specific collection items - are you OK that the processing would just skip these (maybe with some info log fr verbose output)?

---

I also have one more general question. I am testing gatsby v4 compatibility (#192), which is currently in alpha. What do you think about upgrading to the gatsby v4 with your project - my preliminary testing looks like there might not be any breaking changes from the kontent gatsby source plugin point of view.

[anthonyshughes]

@Simply007, @bwlng,

Yes I agree that it would make sense to create a more generic solution to allow filtering on all the system properties rather than just collection.

Cross collection Links
From our point of view, we accept that linking to a collection that is filtered would cause an issue with links. Most of our editors only have access to the collections that are relevant to their site so this shouldn't be a problem for us. We would make it part of the editor training that they can't link to items from other collections for the couple of editors that do have access to multiple sites. I can see it could cause confusion for other customers if they are not aware of this restriction.

Webhook handling
Filtering webhooks by collection is a feature that I would like to see implemented within Kontent. We currently use an AWS Lambda function to receive the webhooks from Kontent and then redistribute them to our Gatsby Cloud sites depending on the collection the webhooks are for. This stops us building sites that do not have any changes. So not a problem for us specifically but again could be an issue for other customers.

I think that the potential issues that could be caused by implementing this feature could also be a problem for anyone using the Delivery Rest API. If these points are documented via the API filtering section then it shouldn't cause too many problems.

Gatsby V4
We will try upgrading and see what happens.

[Simply007]

@bwlng @anthonyshughes

Alright,

how about trying the lean approach?

I will implement the feature, release the feature in alpha as experimental, you will test it out and if everything is OK, I can include it as an experimental feature in the standard release (and add tests for it). After some adoption/testing time, we can continue to push it as standard configuration property in one of the upcoming major/minor releases.

What do you think?

[anthonyshughes]

@bwlng @anthonyshughes

Alright,

how about trying the lean approach?

I will implement the feature, release the feature in alpha as experimental, you will test it out and if everything is OK, I can include it as an experimental feature in the standard release (and add tests for it). After some adoption/testing time, we can continue to push it as standard configuration property in one of the upcoming major/minor releases.

What do you think?

We are happy to go with this approach and test it out.

[anthonyshughes]

@Simply007,

We have upgraded to V4 (4.0.0-alpha-9689ff.5) and it hasn't caused any noticable issues. Seems to have sped up page queries which is nice.

[Simply007]

@Simply007,

We have upgraded to V4 (4.0.0-alpha-9689ff.5) and it hasn't caused any noticable issues. Seems to have sped up page queries which is nice.

Great to hear! I still need to release the kontent gatsby source plugin to ensure that the source plugin is correctly using Gatsby v4.

Out of interest: Could you share the numbers for the time boost?

[Simply007]

@bwlng @anthonyshughes
Alright,
how about trying the lean approach?
I will implement the feature, release the feature in alpha as experimental, you will test it out and if everything is OK, I can include it as an experimental feature in the standard release (and add tests for it). After some adoption/testing time, we can continue to push it as standard configuration property in one of the upcoming major/minor releases.
What do you think?

We are happy to go with this approach and test it out.

I have released version 7.3.0-alpha.0 - could you test it out?

I have described options in the Available options section - so for you, the value could be system.collection=<COLLECTION_CODENAME>.

[anthonyshughes]

Out of interest: Could you share the numbers for the time boost?

This is only from a local single clean (no cache) build for each version so not exactly scientific but a definite improvement.
V3
success run page queries - 1470.570s - 11685/11685 7.95/s
V4
success run queries in workers - 908.137s - 11685/11685 12.87/s

[anthonyshughes]

I have released version 7.3.0-alpha.0 - could you test it out?

I have described options in the Available options section - so for you, the value could be system.collection=<COLLECTION_CODENAME>.

We have tested this using a single collection as the filter and also mutliple (system.collection[in]=austria,default) and it works perfectly. Not only has it reduced the time to source and transform nodes but it has also reduced the time to run the page queries. I assume because the page queries now have less nodes to sort through.

[Simply007]

Perfect @anthonyshughes!

I will keep this version out like that for a while so that you can send your feedback if you have some.
Then I will probably release a new version of the source plugin targeting gatsby v4 (#192) once it is out and stable.
Finally, I will include this feature as experimental in the release/or maybe include it as standard into the release if I have time to come up with a more robust solution.

[bwlng]

@Simply007 I tested 7.3.0-alpha.0 by filtering our archived items

  ...
  experimental: {
    additionalItemFilterParams: `system.workflow_step[neq]=archived`,
  }
  ...

Works as expected, thanks so much! I can't think of any filters outside of the system properties we would need to filter.

• Cross collection Links and Cross Links that aims to filtered-out item – if these are being filtered out to create individual sites, it makes sense to treat them as external links and train editors to handle them that way
• Webhook handling – this won't impact our project, but agree it would be nice to have additional options within Kontent for filtering webhooks based on criteria. We currently filter webhooks in the app that receives them as well

[Simply007]

Thank you both for the feedback @bwlng @anthonyshughes!

May I ask you for one more favor? Could you change the collection/workflow step to the one that is filtered in and vice versa?

I was wondering how does it behaves if you have the update webhook configured time-vise as well as for behavior itself. I guess it should remove/add item, but better to double-check.

[Simply007]

Hey @bwlng and @anthonyshughes,

I have released v8.0.0-beta.0 with gatsby v4 support + filtering model feature:

• https://www.npmjs.com/package/@kentico/gatsby-source-kontent/v/8.0.0-beta.0
• https://www.npmjs.com/package/@kentico/gatsby-kontent-components/v/8.0.0-beta.0

Feel free to test it - feedback is welcome!

[Simply007]

Version 8.0.0-beta.1 of packages:

• https://www.npmjs.com/package/@kentico/gatsby-source-kontent/v/8.0.0-beta.1
• https://www.npmjs.com/package/@kentico/gatsby-kontent-components/v/8.0.0-beta.1

Contains only deprecation fixes - functionality should not be affected.

[anthonyshughes]

@Simply007,
I have been using gatsby-source-kontent/v/8.0.0-beta.1 and gatsby-kontent-components/v/8.0.0-beta.1 for over a week now and all seems fine so far. No issues to report.

[bwlng]

@Simply007 Thanks! I have been using the filtering option in v8 and have no issues to report with this feature. Works great 👍

[Simply007]

I am closing the issue. Feature is released in https://github.com/Kentico/kontent-gatsby-packages/releases/tag/8.0.0