Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read_sas does not handle numeric variables stored with fewer than 8 bytes in SAS datasets #21616

Closed
sasutils opened this issue Jun 24, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@sasutils
Copy link

commented Jun 24, 2018

This problem has been reported twice on stackoverflow, but I cannot find any issue raised here to address it.

https://stackoverflow.com/questions/49059421/pandas-fails-with-correct-data-type-while-reading-a-sas-file
https://stackoverflow.com/questions/51005244/how-can-i-preserve-the-data-type-of-a-column-when-using-pandas-read-sas/51005427#51005427

read_sas() is not handling SAS numeric values that are stored using fewer than the full 8 bytes required for floating point numbers. It appears that instead of padding the short values with binary zeros that read_sas is using some other, perhaps randomly chosen, bytes to fill out the values.

In the first stackover flow example you can see that SAS has stored the value of the number 8. This would be represented in IEEE floating point as the 8 bytes represented by the hex string '40 20 00 00 00 00 00 00'. In the sas7bdat file SAS stored only the 3 most significant bytes. Instead of padding it with 5 bytes of zeros before converting it it looks like read_sas padded it with bytes represented by the hex codes '06 07 80 FD C1' instead. So that instead of reading the value as 8 it was read as 8.00046 .

@sasutils sasutils changed the title read_sas does not handle variables stored with fewer than 8 bytes in SAS datasets read_sas does not handle numeric variables stored with fewer than 8 bytes in SAS datasets Jun 24, 2018

@gfyoung

This comment has been minimized.

Copy link
Member

commented Jun 25, 2018

troels added a commit to troels/pandas that referenced this issue Sep 9, 2018

BUG: Make sure that sas7bdat parsers memory is initialized to 0 (pand…
…as-dev#21616)

Memory for numbers in sas7bdat-parsing was not initialized properly to 0.
For sas7bdat files with numbers smaller than 8 bytes this made the
least significant part of the numbers essentially random.
Fix it by initializing memory correctly.

troels added a commit to troels/pandas that referenced this issue Sep 11, 2018

BUG: Make sure that sas7bdat parsers memory is initialized to 0 (pand…
…as-dev#21616)

Memory for numbers in sas7bdat-parsing was not initialized properly to 0.
For sas7bdat files with numbers smaller than 8 bytes this made the
least significant part of the numbers essentially random.
Fix it by initializing memory correctly.

troels added a commit to troels/pandas that referenced this issue Sep 12, 2018

BUG: Make sure that sas7bdat parsers memory is initialized to 0 (pand…
…as-dev#21616)

Memory for numbers in sas7bdat-parsing was not initialized properly to 0.
For sas7bdat files with numbers smaller than 8 bytes this made the
least significant part of the numbers essentially random.
Fix it by initializing memory correctly.

@jreback jreback added this to the 0.24.0 milestone Sep 15, 2018

jreback added a commit that referenced this issue Sep 15, 2018

@sasutils

This comment has been minimized.

Copy link
Author

commented Sep 15, 2018

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.