Aggregator for csv building

[Jammy2211]

You should first read this issue the AggregatorPng, as the general design and idea is the same:

#1086

Use Case:

Similar to .png splicing, it was common for me to want to view the numerical results of lens modeling in a single
.csv file, rather than navigating the output folder to find the information I needed.

Implementation:

Here is an example of the .csv file for the 2 example images on the agg_png_csv URL above:

https://github.com/Jammy2211/autolens_workspace_test/blob/main/agg_png_csv/result.csv

This .csv file contains headers which come from different parts of the dataset and output folders, thus navigating
the folders to find the information requires combining information from multiple files.

Here is the example .csv maker python file I was using, which is a bit of a mess, but works:

https://github.com/Jammy2211/autolens_workspace_test/blob/main/agg_png_csv/csv_make.py

AggregatorCSV

I am picturing an AggregatorCSV class that would take the output folder and help build the .csv file.

Using Output Folder

For the example on the workspace, all information used to build the .csv file comes from the info.json and
result.json files in the dataset folder.

So you would just need to write an AggregatorCSV that navigates the dataset folder and loads the info.json and
result.json files, and produces the .csv file.

My .csv building never used the output folder, but this was because I wrote a quite complicated pipeline (in a hurry)
which output all results to the dataset folder.

I think the general use case would be that the output folder is used to extract all information whenever possible.

For example, the einstein_radius_max_lh is stored here:

Where samples_summary.json has all the information on the model and instance, with an einstein radius at this part:

    "median_pdf_sample": {
            "type": "instance",
            "class_path": "autofit.non_linear.samples.sample.Sample",
            "arguments": {
                "log_likelihood": 21775.911731379132,
                "log_prior": 1.4651345875102018,
                "weight": 6.53867998265804e-05,
                "kwargs": {
                    "type": "dict",
                    "arguments": {
                        "galaxies.lens.bulge.profile_list.59.centre.centre_0": -0.07025111305343998,
                        "galaxies.lens.bulge.profile_list.59.centre.centre_1": -0.02258690706633907,
                        "galaxies.lens.bulge.profile_list.29.ell_comps.ell_comps_0": 0.250115244578848,
                        "galaxies.lens.bulge.profile_list.29.ell_comps.ell_comps_1": -0.18539079820818452,
                        "galaxies.lens.bulge.profile_list.59.ell_comps.ell_comps_0": 0.05644153124215467,
                        "galaxies.lens.bulge.profile_list.59.ell_comps.ell_comps_1": -0.16025101750452186,
                        "galaxies.source.bulge.profile_list.19.centre.centre_0": -0.02911164136961475,
                        "galaxies.source.bulge.profile_list.19.centre.centre_1": 0.16875116292548728,
                        "galaxies.source.bulge.profile_list.19.ell_comps.ell_comps_0": -0.3747737836275342,
                        "galaxies.source.bulge.profile_list.19.ell_comps.ell_comps_1": -0.191211048679011,
                        "galaxies.lens.mass.ell_comps.ell_comps_0": 0.12954201123863499,
                        "galaxies.lens.mass.ell_comps.ell_comps_1": -0.09446519305629113,
                        "galaxies.lens.mass.einstein_radius": 0.8099997493702147,
                        "galaxies.lens.shear.gamma_1": -0.04293296870204821,
                        "galaxies.lens.shear.gamma_2": 0.10475246785931866
                    }
                }
            }
        },

Therefore an API which allows the user to use the model_path to choose what the .csv headers are would be ideal, something like:

agg = AggregatorCSV.from_directory(
    directory=path.join("output),
)

agg.add_column(
    folder=source_lp[1],
    name="einstein_radius_max_lh",
    model_path="galaxies.lens.mass.einstein_radius",
)

Errors

Note that samples_summary.json also stores errors on parameters, so the API above should be extended to specify the
error on the parameter, e.g.:

agg.add_column(
    folder=source_lp[1],
    name="einstein_radius_max_lh",
    model_path="galaxies.lens.mass.einstein_radius",
    error="errors_at_sigma_3", # this string is in the `samples_summary.json` file
)

Latent Variable API

The AggregatorCSV should also support the latent variable API, so that the user can use the model_path to access the
latent variables of the model.

The example GitHub repo does not have latent variables, but the AggregatorCSV should be designed to support them,
as they will just be in a latent_summary.json file analogous to the samples_summary.json file.

Errors should also be supported for latent variables.

Manual Function API

From samples_summary.json the AggregatorCSV can create an instance of the maximum likelihood or median PDF model.

A user may want to compute a quantity from this instance and add it to the .csv file. This value may be something
you could add as a latent variable, but lets pretend the user forgot to add it before running the pipeline or doesnt
want loads of latent variables in the code.

The following API could allows the user to do this:

agg = AggregatorCSV.from_directory(
    directory=path.join("output),
)

def einstein_radius_x2_from(instance):
    einstein_radius = instance.galaxies.lens.mass.einstein_radius
    return einstein_radius * 2.0

agg.add_column(
    folder=source_lp[1],
    name="einstein_radius_x2_max_lh",
    latent_func=einstein_radius_x2_from,
    use_max_lh_instance=True, # as opposed to the median PDF instance
)

Manual Function with Samples API

The example above uses samples_summary.json to create an instance of the maximum likelihood or median PDF model.
It does not use the full set of non-linear search samples and therefore cannot provide an error estimate on the
quantity computed.

The samples are fully included in samples.json and the AggregatorCSV should support the following API to compute
a quantity from the samples and add it to the .csv file, which can then include an error estimate:

agg = AggregatorCSV.from_directory(
    directory=path.join("output),
)

def einstein_radius_x2_via_samples_from(samples):

    random_draws = 50

    einstein_radius_x2_list = []
    
    for i in range(random_draws):

        instance = samples.draw_randomly_via_pdf()
    
        ell_comps = instance.galaxies.lens.mass.ell_comps
    
        einstein_radius_x2 = al.convert.einstein_radius_x2_from(ell_comps=ell_comps)
    
        einstein_radius_x2_list.append(einstein_radius_x2)
    
    median_einstein_radius_x2, lower_einstein_radius_x2, upper_einstein_radius_x2 = af.marginalize(
        parameter_list=einstein_radius_x2_list,
        sigma=3.0,
    )

    return median_einstein_radius_x2, lower_einstein_radius_x2, upper_einstein_radius_x2
    
agg.add_x3_columns_with_errors(
    folder=source_lp[1],
    name="einstein_radius_x2",
    samples_func=einstein_radius_x2_via_samples_from,
)

Missing .json Files

A user can disable the output of samples.json, so the AggregatorCSV should raise an warning if the user tries to use
the samples_func API and the samples.json file is not present, and leave blank the column in the .csv file.

[Jammy2211]

I have begun writing the workspace example using the CSV aggregator:

https://github.com/Jammy2211/autogalaxy_workspace/blob/main/scripts/results/examples/csv_make.py

Tasks for tomororw:

Tuples dont work

The following works:

agg_csv.add_column(argument="galaxies.galaxy.bulge.effective_radius")

The following does not:

agg_csv.add_column(argument="galaxies.galaxy.bulge.centre.centre_0")

Support maximum log likelihood:

The following part of the doc is not supported yet, with the expectation this will come from samples_summary.json

"""
__Maximum Likelihood Values__

We can also output the maximum likelihood values of each parameter to the .csv file, using the `use_max_log_likelihood`
input.
"""
agg_csv = af.AggregateCSV(aggregator=agg)

agg_csv.add_column(
    argument="galaxies.galaxy.bulge.effective_radius",
    name="bulge_effective_radius_max_lh",
    use_max_log_likelihood=True,
)

agg_csv.save(path="csv_simple_max_likelihood.csv")

Support Errors

The following part of the doc is not supported yet, with the expectation this will come from samples_summary.json

agg_csv = af.AggregateCSV(aggregator=agg)

agg_csv.add_column(
    argument="galaxies.galaxy.bulge.effective_radius",
    name="bulge_effective_radius_sigma",
    use_values_at_sigma=3.0,
)

agg_csv.save(path="csv_simple_errors.csv")

Support Latent Variables

The sample script has been extended to output a latent variable called example_latent, which is output to hard-disk correctly (e.g. there is a latent_summary.json file).

Currently larent_summary.json goes in the files folder but not the files/latent folder, can you make it output to files/latent?

However, I can't add it as a column to the .csv:

agg_csv = af.AggregateCSV(aggregator=agg)

agg_csv.add_column(
    argument="example_latent",
)

agg_csv.save(path="csv_example_latent.csv")

Computed Columns:

There is support for this in the Aggregator, but using it raises an exception:

"""
__Computed Columns__

We can also add columns to the .csv file that are computed from the median PDF instance values of the model.

To do this, we write a function which is input into the `add_computed_column` method, where this function takes the
median PDF instance as input and returns the computed value.

Below, we add a trivial example of a computed column, where the value is twice the sersic index of the bulge.
"""
def sersic_index_x2_from(instance):

    return 2.0 * instance.galaxies.galaxy.bulge.sersic_index

agg_csv = af.AggregateCSV(aggregator=agg)

agg_csv.add_computed_column(
    name="bulge_sersic_index_x2_computed",
    compute=sersic_index_x2_from,
)

agg_csv.save(path="csv_computed_columns.csv")

Allow a user to input a whole list to add a column:

"""
__Column List__

We can add a list of values to the .csv file, provided the list is the same length as the number of model-fits
in the aggregator.

A useful example of this would be adding the name of every dataset to the .csv file in a column on the left,
which would allow you to know which dataset each row corresponds to.

To make this list, we use the `Aggregator` to loop over the `search` objects and extract their `unique_tag`'s, which 
when we fitted the model above used the dataset names. This API can also be used to extract the `name` or `path_prefix`
of the search and build an informative list for the names of the subplots.

We then pass this list to the `add_column` method, which will add a column to the .csv file.
"""
# unique_tag_list = [search.unique_tag for search in agg.values("search")]

# agg_csv = af.AggregateCSV(aggregator=agg)

# agg_csv.add_column(
#     argument=unique_tag_list,
#     name="dataset_name",
# )

# agg_csv.save(path="csv_simple_dataset_name.csv")

Future Tasks:

This will do for now, but the following extensions will be priority once the above all works:

• Extend latent variable API to support max LH and error values.
• Extend computed column API to support max LH values (not just median PDF).
• Extend computed column API to support use of samples.json.

[Jammy2211]

I have tested and updated the example against the new PR which fixes a lot of the code:

https://github.com/Jammy2211/autogalaxy_workspace/blob/main/scripts/results/examples/csv_make.py

I'll note below whats still not supported for future reference:

Errors:

"""
__Errors__

We can also output PDF values at a given sigma confidence of each parameter to the .csv file, using 
the `use_values_at_sigma` input and specifying the sigma confidence.

Below, we add the values at 3.0 sigma confidence to the .csv file, in order to compute the errors you would 
subtract the median value from these values.

The method below adds two columns to the .csv file, corresponding to the values at the lower and upper sigma values.
"""
# agg_csv = af.AggregateCSV(aggregator=agg)
#
# agg_csv.add_column(
#     argument="galaxies.galaxy.bulge.effective_radius",
#     name="bulge_effective_radius_sigma",
#     use_values_at_sigma=3.0,
# )
#
# agg_csv.save(path="csv_simple_errors.csv")

Column List:

"""
__Column List__

We can add a list of values to the .csv file, provided the list is the same length as the number of model-fits
in the aggregator.

A useful example of this would be adding the name of every dataset to the .csv file in a column on the left,
which would allow you to know which dataset each row corresponds to.

To make this list, we use the `Aggregator` to loop over the `search` objects and extract their `unique_tag`'s, which 
when we fitted the model above used the dataset names. This API can also be used to extract the `name` or `path_prefix`
of the search and build an informative list for the names of the subplots.

We then pass this list to the `add_column` method, which will add a column to the .csv file.
"""
# unique_tag_list = [search.unique_tag for search in agg.values("search")]

# agg_csv = af.AggregateCSV(aggregator=agg)

# agg_csv.add_column(
#     argument=unique_tag_list,
#     name="dataset_name",
# )

# agg_csv.save(path="csv_simple_dataset_name.csv")

Latent Variables

"""
__Latent Variables__

Latent variables are not free model parameters but can be derived from the model, and they are described fully in
?.

This example was run with a latent variable called `example_latent`, and below we show that this latent variable
can be added to the .csv file using the same API as above.
"""
# agg_csv = af.AggregateCSV(aggregator=agg)
#
# agg_csv.add_column(
#     argument="example_latent",
# )
#
# agg_csv.save(path="csv_example_latent.csv")

Note that you did add latent variable support, but it gives this error:

Traceback (most recent call last):
  File "/mnt/c/Users/Jammy/Code/PyAuto/autogalaxy_workspace/scripts/results/examples/csv_make.py", line 242, in <module>
    agg_csv.save(path="csv_example_latent.csv")
  File "/mnt/c/Users/Jammy/Code/PyAuto/PyAutoFit/autofit/aggregator/aggregate_csv.py", line 298, in save
    writer.writerow(row.dict())
  File "/mnt/c/Users/Jammy/Code/PyAuto/PyAutoFit/autofit/aggregator/aggregate_csv.py", line 179, in dict
    row[column.name] = column.value(self)
  File "/mnt/c/Users/Jammy/Code/PyAuto/PyAutoFit/autofit/aggregator/aggregate_csv.py", line 62, in value
    return kwargs[self.path]
KeyError: ('example_latent',)

Computed Columns

This works, but could you update it so that the user written function receives both sampels_summary and samples, as the latter may not always be availble (so could be input as None):

def sersic_index_x2_from(samples_summary, samples):

    instance = samples.median_pdf()

    return 2.0 * instance.galaxies.galaxy.bulge.sersic_index

Future Tasks:

This will do for now, but the following extensions will be priority once the above all works:

Extend latent variable API to support max LH and error values.
Extend computed column API to support max LH values (not just median PDF).
Extend computed column API to support use of samples.json.

[Jammy2211]

The CSV making is pretty much working for science, the priorities are as follows:

Error Support

"""
__Errors__

We can also output PDF values at a given sigma confidence of each parameter to the .csv file, using 
the `use_values_at_sigma` input and specifying the sigma confidence.

Below, we add the values at 3.0 sigma confidence to the .csv file, in order to compute the errors you would 
subtract the median value from these values.

The method below adds two columns to the .csv file, corresponding to the values at the lower and upper sigma values.
"""
# agg_csv = af.AggregateCSV(aggregator=agg)
#
# agg_csv.add_column(
#     argument="galaxies.galaxy.bulge.effective_radius",
#     name="bulge_effective_radius",
#     use_values_at_sigma=3.0,
# )
#
# agg_csv.save(path="csv_simple_errors.csv")

The errors should first try and use samples_summary.json, and if the sigma value is not available use samples.csv to compute it.

The columns would be called bulge_effective_radius_upper_3_sigma and bulge_effective_radius_lower_3_sigma.

Multi Column addition via add_column:

After adding the API above, adding columns for the median PDF model, max lh model and errors and 1 and 3 sigma would be as follows:

agg_csv.add_column(
    argument="galaxies.lens.mass.einstein_radius",
    name="mass_einstein_radius",
    use_max_log_likelihood=True,
)

agg_csv.add_column(
    argument="galaxies.lens.mass.einstein_radius",
    name="mass_einstein_radius_max_lh",
    use_max_log_likelihood=True,
)

agg_csv.add_column(
    argument="galaxies.lens.mass.einstein_radius",
    name="mass_einstein_radius_sigma_3",
    use_values_at_sigma=3.0,
)

agg_csv.add_column(
    argument="galaxies.lens.mass.einstein_radius",
    name="mass_einstein_radius_sigma_1",
    use_values_at_sigma=1.0,
)

We can make the API concise such that 6 colums are produced for the following single line of code:

agg_csv.add_column(
    argument="galaxies.lens.mass.einstein_radius",
    name="mass_einstein_radius",
    use_median_pdf=True, # default True
    use_max_log_likelihood=True, # default False
    use_values_at_sigma=[3.0, 1.0]
)

The 6 column headers are:

• mass_einstein_radius.
• mass_einstein_radius_max_lh.
• mass_einstein_radius_upper_3_sigma.
• mass_einstein_radius_lower_3_sigma.
• mass_einstein_radius_upper_1_sigma.
• mass_einstein_radius_lower_1_sigma.

The prefixes max_lh, upper_3_sigma etc I guess should just be default values, maybe they are optional inputs to add column?

Extend add_computed_column

The following example of add_computed_column works:

agg_csv = af.AggregateCSV(aggregator=agg)

def einstein_radius_x2_from(samples):
    instance = samples.median_pdf()

    return 2.0 * instance.galaxies.lens.mass.einstein_radius

agg_csv.add_computed_column(
    name="bulge_einstein_radius_x2_computed",
    compute=einstein_radius_x2_from,
)

agg_csv.save(path=workflow_path / "csv_computed_columns.csv")

Can we extend this functionality so that it returns a dictionary, where the keys of the dictionary are the column names and values are the value, so that we can include, for example, errors:

agg_csv = af.AggregateCSV(aggregator=agg)

def einstein_radius_x2_from(samples, sample_summary):
    instance = samples.median_pdf()

    upper_3_sigma =  instance.galaxies.lens.mass.einstein_radius * 4.0
    lower_3_sigma =  instance.galaxies.lens.mass.einstein_radius * 0.25

    return {
           "einstein_radius_x2" : 2.0 * instance.galaxies.lens.mass.einstein_radius,
           "einstein_radius_x2_upper_3_sigma" : upper_3_sigma,
           "einstein_radius_x2_lower_3_sigma" : lower_3_sigma
   }

agg_csv.add_computed_column(
    compute=einstein_radius_x2_from, # name is removedfrom this function and replaced with dict keys
)

agg_csv.save(path=workflow_path / "csv_computed_columns.csv")

Note also that both samples and samples_summary come into the above function.