Fixes s390x byteswapping issues #6183
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6183 +/- ##
==========================================
+ Coverage 56.95% 57.04% +0.08%
==========================================
Files 506 506
Lines 30467 30965 +498
Branches 4592 4599 +7
==========================================
+ Hits 17353 17663 +310
- Misses 12285 12477 +192
+ Partials 829 825 -4 ☔ View full report in Codecov by Sentry. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # Raw_bytes support: using frombuffer. | ||
| raw_data = tensor.raw_data | ||
| if sys.byteorder == "big": | ||
| # Convert endian from little to big | ||
| raw_data = np.frombuffer(raw_data, dtype=np_dtype).byteswap().tobytes() | ||
| raw_data = np.frombuffer(raw_data, dtype=storage_np_dtype).byteswap().tobytes() |
Check failure
Code scanning / lintrunner
MYPY/call-overload Error
| @@ -288,6 +287,10 @@ | |||
|
|
|||
| return np.frombuffer(raw_data, dtype=np_dtype).reshape(dims) # type: ignore[no-any-return] | |||
|
|
|||
| # dtype for raw data is determined by the storage_dtype | |||
| storage_np_dtype = helper.tensor_dtype_to_np_dtype( | |||
Check failure
Code scanning / lintrunner
MYPY/assignment Error
| @@ -1563,6 +1563,16 @@ | |||
| """ | |||
| return mapping.TENSOR_TYPE_MAP[tensor_dtype].storage_dtype | |||
|
|
|||
| def tensor_dtype_to_storage_np_dtype(tensor_dtype: int) -> int: | |||
| """Convert a TensorProto's data_type to corresponding data_type for raw storage. | |||
|
|
|||
Check warning
Code scanning / lintrunner
RUFF/W293 Warning
|
|
||
| Args: | ||
| tensor_dtype: TensorProto's data_type | ||
|
|
Check warning
Code scanning / lintrunner
RUFF/W293 Warning
| @@ -1563,6 +1563,16 @@ | |||
| """ | |||
| return mapping.TENSOR_TYPE_MAP[tensor_dtype].storage_dtype | |||
|
|
|||
| def tensor_dtype_to_storage_np_dtype(tensor_dtype: int) -> int: | |||
| """Convert a TensorProto's data_type to corresponding data_type for raw storage. | |||
|
|
|||
Check warning
Code scanning / lintrunner
EDITORCONFIG-CHECKER/editorconfig Warning
|
|
||
| Args: | ||
| tensor_dtype: TensorProto's data_type | ||
|
|
Check warning
Code scanning / lintrunner
EDITORCONFIG-CHECKER/editorconfig Warning
| Returns: | ||
| numpy's data_type for the raw storage | ||
| """ | ||
| return mapping.TENSOR_TYPE_MAP[tensor_dtype].storage_np_dtype |
Check failure
Code scanning / lintrunner
MYPY/return-value Error
Description
Updates the mapping of the datatypes to also indicate the element size when stored as a raw tensor. This is separate from the storage dtype, which appears to correspond to the type used when the data is an attribute.
This also updates the testcases which use make_tensor with raw=True to perform byteswapping on big endian systems if the datatype is at least 2 bytes.
Motivation and Context
This fixes the failures identified in issue #6181 as well as a handful of other related s390x specific failures not otherwise identified in the issue. This will be one step closer to a clean testsuite on s390x.