Registering components is sometimes taking too long to show up in the catalog

[dayadev]

Expected Behavior

When I register a new component as per this documentation, it show up in the catalog as soon as its registered.

Actual Behavior

Taking too long for the registered entity to show up in the catalog. sometimes from hours to days to never. When registering, it seems like it's successful without any errors.

Your Environment

• Browser Information: Any known browsers
• Database: postgres
• Version Control: gitlab

[dayadev]

Does it happen if there are huge number of entities like 50k entities?

[freben]

Yeah, but it's still a surprisingly long time. I wonder if your catalog is so overburdened, that its processing etc is piled up in an endless backlog and can't make progress fast enough.

Could it be that the database server is underprovisioned? Is there just a single backend instance handling all catalog load?

[mballoni]

Also facing this and our org is not that big.

I can see the logs:

{"documentType":"software-catalog","level":"info","message":"Collating documents for software-catalog via DefaultCatalogCollatorFactory","plugin":"search","service":"backstage","type":"plugin"}
{"level":"info","message":"127.0.0.1 - - [18/Nov/2022:17:35:22 +0000] \"GET /api/catalog/entities?offset=0&limit=500 HTTP/1.1\" 200 - \"-\" \"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)\"","service":"backstage","type":"incomingRequest"}
{"documentType":"software-catalog","level":"info","message":"Collating documents for software-catalog succeeded","plugin":"search","service":"backstage","type":"plugin"}

It is very fast but a few applications are never detected, unless I add them myself.

We also found this start log:

Task worker starting: GitlabDiscoveryEntityProvider:*:refresh, {"version":2,"cadence":"PT30M","timeoutAfterDuration":"PT30M"}

but never the finish nor any error (timeout, etc)

[Edje-C]

Also facing this and our org is not that big.

I can see the logs:

{"documentType":"software-catalog","level":"info","message":"Collating documents for software-catalog via DefaultCatalogCollatorFactory","plugin":"search","service":"backstage","type":"plugin"}
{"level":"info","message":"127.0.0.1 - - [18/Nov/2022:17:35:22 +0000] \"GET /api/catalog/entities?offset=0&limit=500 HTTP/1.1\" 200 - \"-\" \"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)\"","service":"backstage","type":"incomingRequest"}
{"documentType":"software-catalog","level":"info","message":"Collating documents for software-catalog succeeded","plugin":"search","service":"backstage","type":"plugin"}

It is very fast but a few applications are never detected, unless I add them myself.

We also found this start log:

Task worker starting: GitlabDiscoveryEntityProvider:*:refresh, {"version":2,"cadence":"PT30M","timeoutAfterDuration":"PT30M"}

but never the finish nor any error (timeout, etc)

If they're showing up after you register them manually, this could be an issue of discovery and therefore not related to this issue. You can start a discussion in Discord and open a separate issue about this if you believe it is a bug.

[mballoni]

Thank you for your response @Edje-C .
In fact I didn't register it manually, I left for the auto discovery but this is all I've got.

I didn't see how my issue is different from the OP, do you still believe its the case?

On a second thought now that you said it I'm not sure anymore if the OP is trying to do it manually or with auto discovery (mine is the later).

Thank you,

[dayadev]

For me it's for both auto discovery as well as manual registering of entities. In a recent backstage con, this issue has been brought up and someone implemented a whole new entity provider and its called incremental entity provider, I haven't tried this yet but here is a link just in case. ***@***.***/backstage-plugin-incremental-ingestion-backend

…

On Mon, Nov 21, 2022, 11:49 PM Marcelo Balloni ***@***.***> wrote: Thank you for your response @Edje-C <https://github.com/Edje-C> . In fact I didn't register it manually, I left for the auto discovery but this is all I've got. I didn't see how my issue is different from the OP, do you still believe its the case? On a second thought now that you said it I'm not sure anymore if the OP is trying to do it manually or with auto discovery (mine is the later). Thank you, — Reply to this email directly, view it on GitHub <#13834 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJ7XLWMJDIFCOFTLKP2XUDWJO4LFANCNFSM6AAAAAAQUHPBQY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Edje-C]

Thank you for your response @Edje-C . In fact I didn't register it manually, I left for the auto discovery but this is all I've got.

I didn't see how my issue is different from the OP, do you still believe its the case?

On a second thought now that you said it I'm not sure anymore if the OP is trying to do it manually or with auto discovery (mine is the later).

Thank you,

Hi, yeah the OP is entering components manually.

[Edje-C]

For me it's for both auto discovery as well as manual registering of entities. In a recent backstage con, this issue has been brought up and someone implemented a whole new entity provider and its called incremental entity provider, I haven't tried this yet but here is a link just in case. @./backstage-plugin-incremental-ingestion-backend
…
On Mon, Nov 21, 2022, 11:49 PM Marcelo Balloni @.> wrote: Thank you for your response @Edje-C https://github.com/Edje-C . In fact I didn't register it manually, I left for the auto discovery but this is all I've got. I didn't see how my issue is different from the OP, do you still believe its the case? On a second thought now that you said it I'm not sure anymore if the OP is trying to do it manually or with auto discovery (mine is the later). Thank you, — Reply to this email directly, view it on GitHub <#13834 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJ7XLWMJDIFCOFTLKP2XUDWJO4LFANCNFSM6AAAAAAQUHPBQY . You are receiving this because you authored the thread.Message ID: @.***>

I don't know what happened with the link, could you try sending it again, please?

[ndacic]

hey @dayadev Did you manage to resolve this one ? Facing similar issue, except that I do not get any components to appear. I am receiving following warning:

My config looks like this:

catalog:
  providers:
    gitlab:
      gitlabsaas:
        host: gitlab.com
        entityFilename: catalog-info.yaml
        schedule: 
          frequency: { minutes: 5 }
          timeout: { minutes: 1 }

proxy:
  '/gitlabci':
    target: 'https://gitlab.com/api/v4'
    allowedMethods: ['GET']
    headers:
      PRIVATE-TOKEN: '${GITLAB_TOKEN}'

when I use alternate processor, everything works fine:
https://backstage.io/docs/integrations/gitlab/discovery#alternative-processor

#   locations:
#     - type: gitlab-discovery

[benjdlambert]

Look's like the 1 minute timeout that you're providing in the schedule section might not be enough for it to complete the ingestion. Maybe try bumping that to something a little longer?

schedule: 
  frequency: { minutes: 10 }
  timeout: { minutes: 5 }

[ndacic]

thanks @benjdlambert I will try it out.
In the end I went with gitlab-discovery:

catalog:
  rules:
    - allow:
        - Component
        - API
        - Resource
        - System
        - Domain
        - Location
        - Group
        - User
        - Template
  import:
    entityFilename: catalog-info.yaml
  locations:
    - type: gitlab-discovery
      target: https://gitlab.com/test/blob/*/catalog-info.yaml

but for it I cannot configure scheduler, but only in catalog.ts:

  // Increasing processing interval. 
  builder.setProcessingIntervalSeconds(600);

Am I right?

[benjdlambert]

Exactly @ndacic - that's why it's preferred to use the providers instead as they have their own refresh loop.

[ndacic]

Thanks man! I will try it out and let you know...