Scoreset stuck at "being processed"

[kaiyuegithub]

The tep scoresets are stuck for 1 day at:
"Scores and/or counts are being processed. Refresh this page in a few minutes to check on their status."

Unable to publish.

Here are the links to the tmp scoresets:
https://www.mavedb.org/score-sets/tmp:55aba8dd-6c4c-4242-ad78-237421e618b8
https://www.mavedb.org/score-sets/tmp:9ccbc1f6-bf3b-4bc5-8489-70e531f97ec9

[EstelleDa]

Hi Kaiyue @kaiyuegithub , would you like to create the same score set again, then refresh the page after about 1 minute? I tried creating new score sets but didn't get the same problem. It might be because the server didn't get your uploaded scores successfully. Let me know if you still see the "Scores and/or counts are being processed. Refresh this page in a few minutes to check on their status." after clicking the Save button for around 10 minutes.

[EstelleDa]

Btw, did you upload the same score file that you shared with me?

[kaiyuegithub]

Hi Estelle,

The files are not exactly the same but the first several columns remain the same.
I tried re-upload the scoresets, but now I see this error:

Failed to process score and/or count data: variant column 'hgvs_pro' cannot contain numeric data. If there were issues with validation of individual variants, they will appear in the Variants section of this page.

I don't think there is any numeric data in that column.

Table S2.csv
Table S3.csv

[EstelleDa]

Thanks for the files. I'll try it then.

[EstelleDa]

It's an interesting problem. After reading the uploaded file such as Table_S2.csv in function csv_data_to_df, the first few columns move to the right column's position. The original hgvs_nt column's data became an index. Hgvs_nt and hgvs_spice are all NA, hgvs_pro columns data became score's data. This is why getting an error "variant column 'hgvs_pro' cannot contain numeric data".
No idea what leads to this problem yet. I'll keep digging it.

[EstelleDa]

@jstone-dev In src/mavedb/lib/score_sets.py file of MaveDB_API, there is a function called csv_data_to_df.

ingested_df = pd.read_csv(
        filepath_or_buffer=file_data,
        sep=",",
        encoding="utf-8",
        quotechar="'",
        na_values=extra_na_values,
        keep_default_na=True,
        dtype={**{col: str for col in HGVSColumns.options()}, "scores": float},
    )

In this case, if I change quotechar from ' to ", then MaveDB won't throw any error. In my other two test files, the quotechar are also ". I don't know why for other files ' is fine, but in this file ' doesn't work.

[bencap]

@EstelleDa We might consider changing that function call to:

ingested_df = pd.read_csv(
        filepath_or_buffer=file_data,
        sep=",",
        encoding="utf-8",
        index_col=False,
        quotechar="'",
        na_values=extra_na_values,
        keep_default_na=True,
        dtype={**{col: str for col in HGVSColumns.options()}, "scores": float},
    )

By passing index_col as False we avoid inferring that hgvs_nt is the index col and are then able to use Table_S2.csv directly and pass variant validation. I doubt anyone is using a legitimate index column in their data tables. More likely, the first column in a user provided table should be treated as data.