You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The orient="table" option of the JSON interface enforces the TableSchema specification but only partially.
The table below shows the deviations from the specification:
Data
Data type
table-schema
format
table-schema
type
pandas
Specification orient=’table’
pandas
read_json orient=’table’
datetime
default (datetime ISO8601 in UTC)
datetime
datetime64[ns]
ok
number
default
number
float64
ok
integer
default
integer
int64
ok
boolean
default
boolean
bool
ok
string
default
string
object
ok
custom type
default
any (custom type)
category / string
ok
email
email
string
Format not supported
uri
uri
string
Format not supported
binary
Binary (base64 string)
string
Format not supported
uuid
uuid
string
Format not supported
date, time or datetime with parsable format
any (parsable ?)
date, time, datetime
Format partially supported
date, time or datetime with custom format
<PATTERN>
date, time, datetime
Format not supported
duration
default (lexical duration ISO8601)
duration
timedelta64[ns]
Read_json not yet implemented
Json data
default (json)
object
Unsupported
Json array
default (json array)
array
Unsupported
date
default (date ISO8601)
date
Unsupported
time
default (time ISO8601)
time
Unsupported
year
default
year
Unsupported
month
default
yearmonth
Unsupported
Point (string)
default (string “lon, lat”)
geopoint
Unsupported
Point (geojson array)
array (array [lon, lat])
geopoint
Unsupported
Point (json object)
object (eg {"lon": 90, "lat": 45})
geopoint
Unsupported
Geometry (geojson)
default (geojson spec)
geojson
Unsupported
Geometry (topojson)
Topojson (topojson spec)
geojson
Unsupported
Everything (custom type)
(string)
Only ‘any‘ is supported
Feature Description
proposal
To have a simple and reversible solution which respect the TableSchema specification, I propose to add an 'External Type' (as defined in some JSON export e.g. "ExtDtype":"Float32").
This External Type (NTVtype) can be added to the name of DataFrame Columns following NTV format (as defined in Internet-Draft).
Feature Type
Adding new functionality to pandas
Changing existing functionality in pandas
Removing existing functionality in pandas
Problem Description
The
orient="table"
option of the JSON interface enforces the TableSchema specification but only partially.The table below shows the deviations from the specification:
Data type
format
type
Specification
orient=’table’
read_json
orient=’table’
Feature Description
proposal
To have a simple and reversible solution which respect the TableSchema specification, I propose to add an 'External Type' (as defined in some JSON export e.g.
"ExtDtype":"Float32"
).This External Type (NTVtype) can be added to the name of DataFrame Columns following NTV format (as defined in Internet-Draft).
This proposal is included in the PDEP-0012
The table below shows the correspondence between TableSchema format / type and pandas NTVtype / dtype:
Note:
Issues
The proposal resolves or provides an alternative solution for the following issues: #12997, #14358, #16492, #35420, #35464, #36211, #39537, #49585, #50782, #51375, #52595, #53252
Examples
JSON representation
Alternative Solutions
The alternative solutions are to add a new pandas attribute
ext_type
or to extend thedtype
asNTVtype
.Those solutions have important impacts.
Additional Context
-> NTV repository : https://github.com/loco-philippe/NTV
-> PDEP-0012
The text was updated successfully, but these errors were encountered: