[Question] What is and how to escape "RepIdx" in .csv file when converting to Json

[SummerSun]

I have to apologize first that this is not an issue but a question and you do a GREAT JOB of this package!

My question is I want to do the same thing with C#, but when read in the first line, I got so many "RepIdx". I searched around but have no clue what it is.

I uploaded my file and tried the conversion in you WebSite, it works like a charm! I tried to debug into the source code, no escaping related to the "RepIdx" kind of string. Could you please provide some hint here?

[martindrapeau]

Hi,
I'm not sure where you saw RepIdx. Could you provide a screenshot here?
--Martin

[SummerSun]

@martindrapeau Thanks for replying!

This is the the first line of the source csv file I got:

"RecordId","DataModelId","ADDRESS_LINE1","ADDRESS_LINE2","ADDRESS_LINE3","ADDRESS_LINE4","POSTCODE","CITY","COUNTRY","COUNTRY_ISO_CODE","LATITUDE","LONGITUDE","RepIdx","COUNTRY_REGION","RepIdx","COUNTRY_REGION_TYPE","NUTS1","NUTS2","NUTS3","WORLD_REGION","US_STATE","COUNTY","ADDRESS_TYPE","RepIdx","PHONE_NUMBER","RepIdx","FAX_NUMBER","RepIdx","WEBSITE","RepIdx","EMAIL","BVD_ID_NUMBER","BVD_ACCOUNT_NUMBER","RepIdx","NATIONAL_ID","RepIdx","NATIONAL_ID_LABEL","RepIdx","NATIONAL_ID_TYPE","RepIdx","TRADE_REGISTER_NUMBER","RepIdx","VAT_NUMBER","EUROPEAN_VAT_NUMBER","ECB_FVC_ID","ECB_MFI_ID","TIN","LEI","RepIdx","STATISTICAL_NUMBER","RepIdx","COMPANY_ID_NUMBER","RepIdx","NATIONAL_ID_PREVIOUS","RepIdx","NATIONAL_ID_DATE_PREVIOUS","RepIdx","INFORMATION_PROVIDER_ID","RepIdx","INFORMATION_PROVIDER_ID_LABEL","TICKER","ISIN","RepIdx","PREVIOUS_NAME","RepIdx","PREVIOUS_NAME_DATE","RepIdx","AKA_NAME"

There are so many fields with same name "RepIdx". And this is the result from your tool

[
  {
    "RecordId": "***",
    "DataModelId": "***",
    "ADDRESS_LINE1": "***",
    "ADDRESS_LINE2": "",
    "ADDRESS_LINE3": "",
    "ADDRESS_LINE4": "",
    "POSTCODE": 11416,
    "CITY": "RIYADH",
    "COUNTRY": "***",
    "COUNTRY_ISO_CODE": "SA",
    "LATITUDE": "",
    "LONGITUDE": "",
    **"RepIdx": 0,**
    "COUNTRY_REGION": "",
    "COUNTRY_REGION_TYPE": "",
    "NUTS1": "",
    "NUTS2": "",
    "NUTS3": "",
    "WORLD_REGION": "Middle East",
    "US_STATE": "",
    "COUNTY": "",
    "ADDRESS_TYPE": "***",
    "PHONE_NUMBER": "***",
    "FAX_NUMBER": "***",
    "WEBSITE": "***",
    "EMAIL": "***",
    "BVD_ID_NUMBER": "***",
    "BVD_ACCOUNT_NUMBER": "***",
    "NATIONAL_ID": 1010158683,
    "NATIONAL_ID_LABEL": "***",
    "NATIONAL_ID_TYPE": "***",
    "TRADE_REGISTER_NUMBER": ***,
    "VAT_NUMBER": "",
    "EUROPEAN_VAT_NUMBER": "",
    "ECB_FVC_ID": "",
    "ECB_MFI_ID": "",
    "TIN": ***,
    "LEI": "***",
    "STATISTICAL_NUMBER": "",
    "COMPANY_ID_NUMBER": "***",
    "NATIONAL_ID_PREVIOUS": "",
    "NATIONAL_ID_DATE_PREVIOUS": "",
    "INFORMATION_PROVIDER_ID": "***",
    "INFORMATION_PROVIDER_ID_LABEL": "***",
    "TICKER": ***,
    "ISIN": "***",
    "PREVIOUS_NAME": "",
    "PREVIOUS_NAME_DATE": "",
    "AKA_NAME": "***"
  }
]

Everything works so good with your tool, there is no duplicate key named "RepIdx". I am wondering, how did you get rid of the field "RepIdx"?

[martindrapeau]

Hi Summer,

Your CSV has duplicated keys RepIdx throughout. Currently CSVJSON collapses duplicate keys. That is indeed a bug. It should either report an error and ask you to make them unique, or, even better, it should make them unique itself.

Let me find an automatic solution to this problem.

--Martin

[martindrapeau]

Ok, the issue is now resolved. Here is the testcase demonstrating the fix. Notice that '__#' is added when duplicate column headers are detected.
https://csvjson.com/csv2json/d5c3451fcf0b6e1a1d70a4ece2b6424c

Give it another go with your CSV file. Let me know if everything is resolved.

[SummerSun]

Confirmed that it is resolved, I thought it was a feature 😂😂😂

Anyway, thank you! Really good package!