🐛 Bug: Creating a new Project takes several minutes to complete

[Bastlwastl42]

Please confirm if bug report does NOT exists already ?

• I confirm there is no existing issue for this

Steps to reproduce ?

I have a nocodb instance where i develop and debug a lot, so much rinse and repeat for roughly half a year. Creating a new project without any content takes several minutes to complete. This is while using the UI and by using a post request again the Meta API. There is no log about whats taken so long.

Desired Behavior

• unfortunately, the access log is no longer given, so a proper debug/info loggin option would help to understand what exactly is going on
• creating a simple project without any data or tables should not take that long at all

Project Details

Project Info
Node: v16.19.1
Arch: x64
Platform: linux
Docker: true
RootDB: pg
PackageVersion: 0.109.1

Project was created by clicking New Project
Database on postgres 12

Attachments

No response

[wingkwong]

1. does it happen only when you create a new project? how about creating a table or inputting data?
2. what do you mean by "by using a post request again the Meta API"? the request is only triggered from UI right?
3. where is your NC_DB? on cloud? local?

[Bastlwastl42]

re

1. yes, during creatin of the new instance, before there is even the chance to add tables and rows and such inputting data
2. in order to create these projects, i make a post request to '/api/v1/db/meta/projects/' of my noco instance, which is the same thing as done in the UI by clicking on 'new project' from the starting dashboard.
3. The NC_DB is in the container next to the noco_db instance itself, so on the same server, for the noco itself 'local', i guess?

[o1lab]

Sorry, the above description although detailed is confusing for us to understand - the slowness is seen also using our UI or only via API ? If its fine while doing UI and slow from API - we will be unable to help.

Plus as @wingkwong checked in (1) - what about the other operations say creating table / column / view etc ?

[Bastlwastl42]

• any Other Data operation are fine
• Both ways take several minutes, creating through UI AND through directly sending the same request from a script i'm working on.

I made a screen capture where i tried to show as much as possible without disclosing internals. The capture end exactly at the time i would be shown the user management for this new project. It is, to be fair, pretty boring though. ;)

Screen.Recording.2023-08-17.at.12.38.22.mov

[Bastlwastl42]

Also, fun fact, it appears to me that this is only the request that is hanging. While i'm waiting for the script, i can see and select the new project already in the UI. Not sure if there is unfinished business that could break things. Here another capture to show what i meant

Screen.Recording.2023-08-17.at.12.52.55.mov

[wingkwong]

@Bastlwastl42 please DM me at Discord.

[wingkwong]

UPD: Given the stats provided by user, the possible root cause may be from populateMeta. tableList and relationListAll may be the most expensive part. The corresponding optimisation has been conducted in PR6156. Hence, let's wait for the next release to see if it fixes or not.

[dstala]

@Bastlwastl42 Let us know if you still see issue with new release?

[Bastlwastl42]

@dstala @wingkwong I promise i will, but please be patient. Upgrading might take some time with the changes regarding linkage. I'll keep you posted. Meanwhile, thanks for the work 😄

[Bastlwastl42]

@dstala I was finally able to upgrade our nocodb instance to 0.111.4 (i know this no longer the latest, but it's okay)

• the project creation is again at reasonable times, i.e. maybe a second or two and that is totally fine.
• I did the upgrade today, so far runtime is < 1 day, we'll see if the issue reproduces
• during update migration i ran into several issues. The messages on the screen told me that foreign keys are removed, and i hit configured limits of the connected postgres12 DB that is used. The error message from psql was clear enough to hint me to the required configuration changes for the 'max_lock_per_transaction' setting on postgres. This stackoverflow tells you all you need to start and settings these values requires a restart of the PSQL server

I guess i'm fine with having this issue closed, upgrading helps and i'll come back if is see further issues with large projects and tables.