Some data getting mapped is not displaying in detail and list view

[RDmitchell]

SHA: bd59264
Instance: seeddev

See this doc
1213 - Some data getting mapped is not displaying in detail and list view
https://drive.google.com/open?id=1trcdpJ7zGNEN832YJFyv21z597oL8bG5m8VQaDBQZy8

in this folder (folder also has the sample data file)
https://drive.google.com/open?id=0B3fTKpZ9Dx7LZmtYNkExNjEyMms

[RDmitchell]

I think the issues shown in the doc may need to be resolved before releasing V 2.0

[RDmitchell]

This is still an issue -- see the above doc, updated from testing on seeddev with the V2.0 release SHA. Just a few fields that are not either available in the Settings view to select, or if they are there, there is no data displayed when there should be data.

[RDmitchell]

SHA: 9e6e90e
instance: seeddev
browser: chrome incognito
org: LBNL 400
files: Issue 1213 Data file-01.xlsx & Issue 1213 Data file-02.xlsx
Link to folder:
https://drive.google.com/open?id=0B3fTKpZ9Dx7LZmtYNkExNjEyMms

I thought I figured out a way to make the program properly import/display the data with problem fields by renaming the fields but if you read to the very end, you will see my conclusion was actually not correct. Just goes to show what happens when you try to understand / debug based on empirical data. Hopefully someone can look at the behavior I am seeing, and then the code, and make some sense out of it and fix the problem.

Steps to reproduce:

• Import Issue 1213 Data file-01.xlsx
  

• Map fields "Square Footage OPA" and "Gross Floor Area" to the same names as in the file.
  

• Matching Screen -- these fields do not show up in the matching screen
  

• Inventory -- the field names are available in List Settings but no data is displayed.
  

• Delete inventory, data set, start again

• Import same file, this time map problem fields to a different name
  

• Results are the same -- fields don't show up in matching screen
  

• Inventory -- same thing, can select the fields but they don't display data.
  

• Import file: Issue 1213 Data file-02.xlsx - this file has the fields renamed to the new names they were mapped to
  

• Map as shown, ie, keep the mapping as the names in the file
  

• Mapping review screen shows fields and data
  

• Inventory shows fields and data
  

One more test

• Import file: Issue 1213 Data file-03.xlsx -- completely new names for the two problem fields
  
• Mapping to the exact (new) names in the file:
  
• Matching screen -- new field names are not in the matching review screen
  
• Inventory screen -- fields are in List Settings. Program made one bogus Property record for the case where the Jurisdiction TLID was NA, and then that record matched to the Tax Lot record, and that is the only one that has data
  

I have not idea what the program is doing, but need some answers about how to work around this issue.

[RDmitchell]

@nllong -- I am wondering if the problem is that when the data does not match the field type for structured fields then maybe the program doesn't show the data?

I have an example where I mapped a field to "Property Type", although I did assign it to the Tax Lot table, which I would assume would mean it would get put into the Tax Lot extra data field, and therefore would be considered a string (?). And I suspect the Property Type field in the Property table is also considered a string.

However, the data in the original XLS file was format == General, but the data is a number, ie, 500, 115, etc

In the mapping review screen, the data is shown in the Property Type field

But in the final inventory view for Tax Lot, that field does not display any data.

When I reimport and map "Property Type" to "Tax Lot Property Type" which I am assuming will definitely throw it into extra data, the field appears, with the appropriate data.

[RDmitchell]

@nllong -- we need to figure out what is going on with this before the next update, because it will affect users uploading their data.

[RDmitchell]

Working on Atlanta's data and 3 fields mapped to Property table showed up in list settings but without data (when they did have data in the imported file). Changed their field names to uppercase in the original spreadsheet and then the file would import. The original fieldnames were Owner Name, Owner State, and they showed up in the Settings list but didn't display data. I changed those field names in the original file to OWNER Name, OWNER State, etc, (which matched other OWNER fields in the file) and then mapped them to the exact names as the field names, and then the data showed up when selecting those fields in settings.

VERY WEIRD !!!

We need to get this fixed soon because users (and myself) are trying to import real data into the production server and this creates a problem when trying to do that.

[RDmitchell]

Also had an issue that the program automatically mapped Property Code to Property Type. I deleted the inventory and reimported the file, mapping it to Property Code -- the Property Code field was in List Settings but when selected no data displayed. Changed it in the original import file to be all upper case, ie, PROPERTY CODE, then mapped it to PROPERTY CODE (by copying the field on the right to the SEED field column) and now the data shows up when I select PROPERTY CODE in List Settings. This will be a problem for future imports if that field is not changed to PROPERTY CODE in other files imported with that field name.

This has been very frustrating -- I have spent over an hour trying to get the mapping right, and all the data to display properly. Hopefully we can get this fixed soon.

See doc (under mapping section)
https://docs.google.com/a/lbl.gov/document/d/18l87JeJD4w2ikblUWmNGQVdDvckhpluc25H-NaIr1u4/edit?usp=sharing

SHA; .3daf0f8
instance: seedv2 production

[nllong]

oh this is a good one. Looks like the column header has an extra space in it. I'll force it to strip the spaces during column lookup.

[RDmitchell]

Hmmm.. ok, so it's an issue with the headers in the imported data?

Let me check that out in the files that I was having trouble with

Can we strip out leading spaces in headers on import?

[RDmitchell]

I can actually see the space in front of Gross Floor Area in the first screen shot of this issue.

Let me look at the other files and see if they have the same issue.

[nllong]

yeah, it is a headers issue. We do various levels of stripping, but looks like we missed one to make it work right.

[RDmitchell]

instance: seeddev
SHA: aa55bc4
org: LBNL 400

@nllong -- I just put another file for you to test in the 1213 folder
https://drive.google.com/open?id=0B3fTKpZ9Dx7LZmtYNkExNjEyMms

Try importing the file called
1213- 2015 Covered Buildings.xlsx

See this doc
1213 - Another example file

I don't think it's just a space in the beginning of the header. Look at the doc and try importing the file and maybe you will discover some other issue with headers. Thanks.

[RDmitchell]

Was testing for #1128, but Test #3 in the linked doc below shows the problem for this issue ( #1213 ), which I think now @axelstudios can reproduce locally, so not sure if this additional information is needed or useful, but thought I would link to it here anyway, for completeness.

https://drive.google.com/open?id=1f8FWqnza1xPzUhrm_PP0DyboMpgGrNUJ2WxHKLGrRLM

[RDmitchell]

I think this is fixed. I tested several files. So I am closing