Single cell metadata and "global" metadata are used together.

[Arkkienkeli]

We need to separate metadata use for the different types of training.
Single-cell metadata should be used separately in single-cell training.

Global metadata is primarily for profiling.

We need to parameterize the name of the single-cell metadata file.

[Arkkienkeli]

Probably the use of already existing --metadata parameter is a way to go.
In case of profiling and old version of the training the metadata file is searched in inputs/metadata/ and in case of new versions of training in the config['paths']['single_cell_sample']

[Arkkienkeli]

@jccaicedo
There is a question, how we could utilize config for single-cell metadata and how we should generate metadata.

1. The single-cell metadata should include the split_field, because currently it contains either only training set or all single cells without splitting the data into training and validation sets. Probably we should update sampling.py and include split_field. In the latest experiments the splitting was done manually.
2. Then in training routine we can use the information from the config like in the snippet below.

    path = config["paths"]["single_cell_sample"]
    dataset, steps_per_epoch = make_dataset(path, BATCH_SIZE,
                                            single_cell_metadata[
                                                single_cell_metadata[config["train"]["partition"]["split_field"]]
                                                in config["train"]["partition"]["training_values"]],
                                            config,
                                            is_training=True)
    validation_dataset, _ = make_dataset(path, BATCH_SIZE,
                                         single_cell_metadata[
                                             single_cell_metadata[config["train"]["partition"]["split_field"]]
                                             in config["train"]["partition"]["validation_values"]],
                                         config,
                                         is_training=False)

[jccaicedo]

I think we probably should support export only rather than sampling single cells. If someone wants to sample, they can create a new index.csv file with the samples of their interest. Perhaps we should not assume a generic sampling strategy. In fact, after having single cells exported, we can further do other samplings by reorganizing data and annotations on the single-cell metadata file.

So, to answer your questions: let's remove single-cell sampling from DeepProfiler, and let's only keep export (let's do this in another issue and link it here). Second, let's keep the split_field in the single-cell metadata file as you suggest, but only in the export function.

[Arkkienkeli]

Closed by #280