Error: Db is busy: Profile is already in use by another process

[spowell-216]

we are getting the following error message on IModelHost.startup().

Error: Db is busy: Profile [/Application/app/server/src/server/assets/temp/obj-builder/obj-builder-7fcfc978b9-6bg6m/cache/profiles/obj-builder-7fcfc978b9-6bg6m] is already in use by another process

this is a backend only application. Everything runs fine locally on Windows, but when we deploy to AKS is when we are seeing the error.

everything after /temp is a mounted Azure File share.
obj-builder-7fcfc978b9-6bg6m is os.hostname()

code snipet for starting up IModelHost

const imhOptions: IModelHostOptions = {
   appAssetsDir: '/Application/app/server/src/server/assets',
   cacheDir: '/Application/app/server/src/server/assets/temp/obj-builder/obj-builder-7fcfc978b9-6bg6m/cache',
   authorizationClient: valid AuthorizationClient,
   logTileLoadTimeThreshold: IModelHostConfiguration.defaultLogTileLoadTimeThreshold,
   logTileSizeThreshold: IModelHostConfiguration.defaultLogTileSizeThreshold,
   tileContentRequestTimeout: IModelHostConfiguration.defaultTileRequestTimeout,
   tileTreeRequestTimeout: IModelHostConfiguration.defaultTileRequestTimeout,
   profileName: os.hostname()
};

await IModelHost.startup(imhOptions);
IModelHost.applicationId = 'obj-builder';
IModelHost.applicationVersion = "v1.0";

we are using the node:18-bullseye-slim docker image.
and using version ^4.0.6 of the @itwin libraries

let me know if you require any additional information to help troubleshoot the issue.
Thanks.

[kabentley]

You cannot have more than one iTwin.js process running at the same time using the same profile directory. You should choose something other than os.hostname() to make it unique. See this note.

[pratikshan85]

If the profile is already in use, it means they are running multiple itwin.js backend processes on the same machine. They must specify a separate profile (as documented here). Likely they don't want multiple processes, in which case they need to find out why they're accidentally spawning multiple.

[spowell-216]

@kabentley @pratikshan85 thanks for the quick reply. we did see that note for profile and why is we implemented the os.hostname() to get the Kubernetes pod name so we have ensured that they are unique folder path between the different instances.

[kabentley]

somehow you must be starting more than one process per pod.

BTW, if the cacheDir is already unique per process (probably a good idea, depending on what you're application does) you don't need to worry about profileName, since all that does is create a subdirectory below cacheDir. But the error you're seeing can only happen if there are two active processes using the same profile directory.

[kabentley]

I'm sorry to hear that. AFS simply isn't an appropriate solution for a cacheDir. We should try to figure out what's using space in the cacheDir. How large are your tmp drives?

[spowell-216]

I agree AFS or any remote storage should not be used. Is there any way we can set the cachDir size in IModelHostOptions that can be passed along to makeCache. the default '10G' is too big for our Kubernetes pod.

[kabentley]

We will have to add that. I don't know why that's not there now. It won't use 10G anyway now, since you don't have any workspace data to load.

In the meantime, perhaps if you create the profile directory in your code and put a (1s?) wait before calling startup. Maybe that will cause the "busy" error from the AFS "drive" to go away.

[kabentley]

Also, can you try an experiment on your Windows machine where things work? Run your program and do a "dir/s" on the cacheDir after it finishes so we can see what's using the disk space. My guess is its the briefcase cache.

[spowell-216]

thanks for the suggestion, we will try that recommendation in the meantime.