[BUG] Multiprocessing error when using multi-GPU feature extraction

[NaokiThread]

Description

I encountered an issue when attempting to use multi-GPU processing with a feature extractor in my code. The problem occurs because the multiprocessing starts from the beginning of the code, which includes the creation of a project.

The error message I received is:

Copy code
raise OSError(f"A project already exists at {root}")
This error suggests that the code is trying to create a project directory that already exists, likely due to the presence of a "setting.json" file in the project directory.

When the multiprocessing starts from the beginning of the code, it attempts to create the project directory again, resulting in an OSError because the directory already exists.

To resolve this issue, I need guidance on restructuring my code to separate the project creation logic from the multi-GPU processing of the feature extractor. The goal is to ensure that the project creation is executed only once and doesn't interfere with the parallel processing of the feature extractor.

Any suggestions or insights on how to properly handle this scenario would be greatly appreciated. Thank you!

[jamesdolezal]

Does your workflow require that a new project be created from the same script that is executing the feature bag generation? Typically, once a project is created, you should not need to create one again. An example workflow might look like:

Step 1: Create a project

Run code that creates the project, e.g. You should only do this once.

import slideflow as sf
sf.create_project(...)

Step 2: Load the project and do something

In a separate script, load the existing project and generate feature bags, execute, training, etc. Example:

import slideflow as sf
project = sf.load_project(...)
project.generate_feature_bags(...)

[NaokiThread]

Sure! I will do as you instructed me. Thank you!