Issue with downloading database tables

[becca9835]

When downloading data from database tables (both via pandas as well as through external software like Beekeeper Studio) I am getting excess columns and rows filled with numbers that do not seem to have a meaning. This is making it difficult to sort/filter through the data because one column that is meant to have standard values (such as formula_anonymous, where all of the entries should be in the format "ABx") now also has random values such as "-2.1349". I'm not sure if this is due to the way my csv is being converted (i.e. things that aren't supposed to be their own column are being formatted that way due to excess commas?) or if this is intrinsic to the data. I've attached an example of a table I tried to download via Beekeeper Studio (compressed because the original file was too large).
public_workflows_bandstructurecalc_export_2023-07-18_142748.zip

[jacksund]

I'm not sure if this is due to the way my csv is being converted (i.e. things that aren't supposed to be their own column are being formatted that way due to excess commas?) or if this is intrinsic to the data

It's because of the excess commas. The column band_structure_data contains data in the JSON format, and it looks like beekeeper studio is struggling to handle it. When I look inside your csv file, I can see they are storing JSON in a wacky way -- for example

"{""database_id"":390,""database_table"":""StaticEnergy""}"

Using pandas should work fine though... Even something like MyTable.objects.to_dataframe().to_csv("example.csv") should work. Could you share a csv made using that method?

As a fallback, you can also remove json-based columns since you won't want to filter with them in excel anyways. So only give the columns you want:

# I commented out some columns but you can alter this list as you need to
columns_to_include = [
    "id",
    "created_at",
    "updated_at",
    # "source",
    # "structure",
    "nsites",
    "nelements",
    "elements",
    "chemical_system",
    "density",
    "density_atomic",
    "volume",
    "volume_molar",
    "formula_full",
    "formula_reduced",
    "formula_anonymous",
    "started_at",
    "finished_at",
    "total_time",
    "queue_time",
    "workflow_name",
    "workflow_version",
    "computer_system",
    "directory",
    "run_id",
    "corrections",
    # "band_structure_data",
    "nbands",
    "band_gap",
    "is_gap_direct",
    "band_gap_direct",
    "energy_fermi",
    "conduction_band_minimum",
    "valence_band_maximum",
    "is_metal",
    "spacegroup_id",
]
df = MyTable.objects.to_dataframe(columns_to_include)

df.to_csv("example.csv")