Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some data getting mapped is not displaying in detail and list view #1213

Closed
RDmitchell opened this issue Jan 14, 2017 · 14 comments
Closed

Some data getting mapped is not displaying in detail and list view #1213

RDmitchell opened this issue Jan 14, 2017 · 14 comments

Comments

@RDmitchell
Copy link

RDmitchell commented Jan 14, 2017

SHA: bd59264
Instance: seeddev

See this doc
1213 - Some data getting mapped is not displaying in detail and list view
https://drive.google.com/open?id=1trcdpJ7zGNEN832YJFyv21z597oL8bG5m8VQaDBQZy8

in this folder (folder also has the sample data file)
https://drive.google.com/open?id=0B3fTKpZ9Dx7LZmtYNkExNjEyMms

@RDmitchell
Copy link
Author

I think the issues shown in the doc may need to be resolved before releasing V 2.0

@RDmitchell
Copy link
Author

This is still an issue -- see the above doc, updated from testing on seeddev with the V2.0 release SHA. Just a few fields that are not either available in the Settings view to select, or if they are there, there is no data displayed when there should be data.

@RDmitchell
Copy link
Author

SHA: 9e6e90e
instance: seeddev
browser: chrome incognito
org: LBNL 400
files: Issue 1213 Data file-01.xlsx & Issue 1213 Data file-02.xlsx
Link to folder:
https://drive.google.com/open?id=0B3fTKpZ9Dx7LZmtYNkExNjEyMms

I thought I figured out a way to make the program properly import/display the data with problem fields by renaming the fields but if you read to the very end, you will see my conclusion was actually not correct. Just goes to show what happens when you try to understand / debug based on empirical data. Hopefully someone can look at the behavior I am seeing, and then the code, and make some sense out of it and fix the problem.

Steps to reproduce:

  • Import Issue 1213 Data file-01.xlsx
    image

  • Map fields "Square Footage OPA" and "Gross Floor Area" to the same names as in the file.
    image

  • Matching Screen -- these fields do not show up in the matching screen
    image

  • Inventory -- the field names are available in List Settings but no data is displayed.
    image

  • Delete inventory, data set, start again

  • Import same file, this time map problem fields to a different name
    image

  • Results are the same -- fields don't show up in matching screen
    image

  • Inventory -- same thing, can select the fields but they don't display data.
    image

  • Import file: Issue 1213 Data file-02.xlsx - this file has the fields renamed to the new names they were mapped to
    image

  • Map as shown, ie, keep the mapping as the names in the file
    image

  • Mapping review screen shows fields and data
    image

  • Inventory shows fields and data
    image

One more test

  • Import file: Issue 1213 Data file-03.xlsx -- completely new names for the two problem fields
    image
  • Mapping to the exact (new) names in the file:
    image
  • Matching screen -- new field names are not in the matching review screen
    image
  • Inventory screen -- fields are in List Settings. Program made one bogus Property record for the case where the Jurisdiction TLID was NA, and then that record matched to the Tax Lot record, and that is the only one that has data
    image

I have not idea what the program is doing, but need some answers about how to work around this issue.

@RDmitchell
Copy link
Author

RDmitchell commented Feb 1, 2017

@nllong -- I am wondering if the problem is that when the data does not match the field type for structured fields then maybe the program doesn't show the data?

I have an example where I mapped a field to "Property Type", although I did assign it to the Tax Lot table, which I would assume would mean it would get put into the Tax Lot extra data field, and therefore would be considered a string (?). And I suspect the Property Type field in the Property table is also considered a string.

However, the data in the original XLS file was format == General, but the data is a number, ie, 500, 115, etc

In the mapping review screen, the data is shown in the Property Type field

But in the final inventory view for Tax Lot, that field does not display any data.

When I reimport and map "Property Type" to "Tax Lot Property Type" which I am assuming will definitely throw it into extra data, the field appears, with the appropriate data.

@RDmitchell
Copy link
Author

@nllong -- we need to figure out what is going on with this before the next update, because it will affect users uploading their data.

@RDmitchell
Copy link
Author

Working on Atlanta's data and 3 fields mapped to Property table showed up in list settings but without data (when they did have data in the imported file). Changed their field names to uppercase in the original spreadsheet and then the file would import. The original fieldnames were Owner Name, Owner State, and they showed up in the Settings list but didn't display data. I changed those field names in the original file to OWNER Name, OWNER State, etc, (which matched other OWNER fields in the file) and then mapped them to the exact names as the field names, and then the data showed up when selecting those fields in settings.

VERY WEIRD !!!

We need to get this fixed soon because users (and myself) are trying to import real data into the production server and this creates a problem when trying to do that.

@RDmitchell
Copy link
Author

RDmitchell commented Feb 4, 2017

Also had an issue that the program automatically mapped Property Code to Property Type. I deleted the inventory and reimported the file, mapping it to Property Code -- the Property Code field was in List Settings but when selected no data displayed. Changed it in the original import file to be all upper case, ie, PROPERTY CODE, then mapped it to PROPERTY CODE (by copying the field on the right to the SEED field column) and now the data shows up when I select PROPERTY CODE in List Settings. This will be a problem for future imports if that field is not changed to PROPERTY CODE in other files imported with that field name.

This has been very frustrating -- I have spent over an hour trying to get the mapping right, and all the data to display properly. Hopefully we can get this fixed soon.

See doc (under mapping section)
https://docs.google.com/a/lbl.gov/document/d/18l87JeJD4w2ikblUWmNGQVdDvckhpluc25H-NaIr1u4/edit?usp=sharing

SHA; .3daf0f8
instance: seedv2 production

@nllong
Copy link
Member

nllong commented Feb 7, 2017

oh this is a good one. Looks like the column header has an extra space in it. I'll force it to strip the spaces during column lookup.

image

@RDmitchell
Copy link
Author

Hmmm.. ok, so it's an issue with the headers in the imported data?

Let me check that out in the files that I was having trouble with

Can we strip out leading spaces in headers on import?

@RDmitchell
Copy link
Author

I can actually see the space in front of Gross Floor Area in the first screen shot of this issue.

Let me look at the other files and see if they have the same issue.

@nllong
Copy link
Member

nllong commented Feb 7, 2017

yeah, it is a headers issue. We do various levels of stripping, but looks like we missed one to make it work right.

@RDmitchell
Copy link
Author

instance: seeddev
SHA: aa55bc4
org: LBNL 400

@nllong -- I just put another file for you to test in the 1213 folder
https://drive.google.com/open?id=0B3fTKpZ9Dx7LZmtYNkExNjEyMms

Try importing the file called
1213- 2015 Covered Buildings.xlsx

See this doc
1213 - Another example file

I don't think it's just a space in the beginning of the header. Look at the doc and try importing the file and maybe you will discover some other issue with headers. Thanks.

@nllong nllong mentioned this issue Feb 8, 2017
4 tasks
@RDmitchell
Copy link
Author

Was testing for #1128, but Test #3 in the linked doc below shows the problem for this issue ( #1213 ), which I think now @axelstudios can reproduce locally, so not sure if this additional information is needed or useful, but thought I would link to it here anyway, for completeness.

https://drive.google.com/open?id=1f8FWqnza1xPzUhrm_PP0DyboMpgGrNUJ2WxHKLGrRLM

@RDmitchell
Copy link
Author

I think this is fixed. I tested several files. So I am closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants