TF 2.0 crossed_column on Windows fails with SystemError: <built-in function TFE_Py_FastPathExecute> returned a result with an error set

[hsm207]

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 Home
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v1.12.0-9492-g2c319fb415 2.0.0-alpha0
• Python version: 3.6.8 |Anaconda, Inc.| (default, Feb 21 2019, 18:30:04) [MSC v.1916 64 bit (AMD64)]
• Bazel version (if compiling from source): NA
• GCC/Compiler version (if compiling from source): NA
• CUDA/cuDNN version: NA
• GPU model and memory: NA

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
The code snippets works fine on Colab but gives the following error on Windows:

OverflowError: Python int too large to convert to C long
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\IPython\core\interactiveshell.py", line 3296, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-6cfdea12e863>", line 44, in <module>
    demo(feature_column.indicator_column(crossed_feature))
  File "<ipython-input-6-6cfdea12e863>", line 36, in demo
    print(feature_layer(example_batch))
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 660, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\feature_column\feature_column_v2.py", line 473, in call
    self._state_manager)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\feature_column\feature_column_v2.py", line 4391, in get_dense_tensor
    return transformation_cache.get(self, state_manager)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\feature_column\feature_column_v2.py", line 2573, in get
    transformed = column.transform_feature(self, state_manager)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\feature_column\feature_column_v2.py", line 4330, in transform_feature
    transformation_cache, state_manager)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\feature_column\feature_column_v2.py", line 4184, in get_sparse_tensors
    transformation_cache.get(self, state_manager), None)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\feature_column\feature_column_v2.py", line 2573, in get
    transformed = column.transform_feature(self, state_manager)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\feature_column\feature_column_v2.py", line 4145, in transform_feature
    hash_key=self.hash_key)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\ops\sparse_ops.py", line 564, in sparse_cross_hashed
    name=name)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\ops\sparse_ops.py", line 617, in _sparse_cross_internal
    name=name)
  File "C:\Users\user\Anaconda3\envs\tf2\lib\site-packages\tensorflow\python\ops\gen_sparse_ops.py", line 1372, in sparse_cross
    "internal_type", internal_type)
SystemError: <built-in function TFE_Py_FastPathExecute> returned a result with an error set

Executing:

import sys
sys.maxsize

gives:
9223372036854775807

Describe the expected behavior
Same output as running on Colab:

Code to reproduce the issue

import tensorflow as tf
import pandas as pd

from tensorflow import feature_column
from tensorflow.keras import layers
from sklearn.model_selection import train_test_split

URL = 'https://storage.googleapis.com/applied-dl/heart.csv'
dataframe = pd.read_csv(URL)
dataframe.head()

train, test = train_test_split(dataframe, test_size=0.2)
train, val = train_test_split(train, test_size=0.2)

def df_to_dataset(dataframe, shuffle=True, batch_size=32):
  dataframe = dataframe.copy()
  labels = dataframe.pop('target')
  ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
  if shuffle:
    ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds


batch_size = 5 # A small batch sized is used for demonstration purposes
train_ds = df_to_dataset(train, batch_size=batch_size)
val_ds = df_to_dataset(val, shuffle=False, batch_size=batch_size)
test_ds = df_to_dataset(test, shuffle=False, batch_size=batch_size)

example_batch = next(iter(train_ds))[0]

# A utility method to create a feature column
# and to transform a batch of data
def demo(feature_column):
  feature_layer = layers.DenseFeatures(feature_column)
  print(feature_layer(example_batch))

age = feature_column.numeric_column("age")
age_buckets = feature_column.bucketized_column(age, boundaries=[18, 25, 30, 35, 40, 45, 50, 55, 60, 65])
thal = feature_column.categorical_column_with_vocabulary_list(
      'thal', ['fixed', 'normal', 'reversible'])

crossed_feature = feature_column.crossed_column([age_buckets, thal], hash_bucket_size=3)
demo(feature_column.indicator_column(crossed_feature))

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

[gadagashwini-zz]

@hsm207 I tried reproducing the issue on my system but the code executed without any error. Can you try once again and let us know if that still gives error. Thanks!

[hsm207]

@gadagashwini yes, I still get the same error. Is there additional information I can provide to help you diagnose this issue?

[TheodoreGalanos]

I have the same issue following the same 'Classify structured data' tutorial. Same characteristics as the original poster, only I'm running tf on GPU.

[oceanlq]

Same here.

[yuyongze]

Same here on 2.0 beta version

[Daqure]

I get the same error about the "Built-in function ... returned a result with an error set" but refers to the function 'TFE_Py_TapeWatch'.
I find that all functions with 'TFE' are defined in pywrap_tensorflow_internal. So I try to import it on tape but the result is the same as before.
Version: tensorflow-gpu== 2.1 beta
Platform: Google Colab

[doctortellini]

I get the same error about the "Built-in function ... returned a result with an error set" but refers to the function 'TFE_Py_TapeWatch'.
I find that all functions with 'TFE' are defined in pywrap_tensorflow_internal. So I try to import it on tape but the result is the same as before.
Version: tensorflow-gpu== 2.1 beta
Platform: Google Colab

I am having the same error... any solutions?

[Akoopie]

I am also having the same error in the "Classify structured data" guide for tf 2.0.
version 2.0.0 beta1
Platform: Jupyter Lab
Windows.

I also had a similar issue on another model and both are listing gen_sparse_ops.py as the last script in the traceback.

[gadagashwini-zz]

I tried on Colab as well on local system with Tensorflow 2.0.0.rc0. It is working as expected, can you please try with latest TF version and check.

PTAL colab gist here and Jupyter notebook gist here
#28846.ipynb.tar.gz
.Let us know how it progresses. Thanks!

[hsm207]

I tried running the code again after upgrading to v2.0.0-rc0-101-gd2d2566eef 2.0.0-rc1 and now the error is gone. Thanks @gadagashwini

[gadagashwini-zz]

Glad it is working.
Closing the issue. Please feel free to reopen if still issue persists. Thanks!

[tensorflow-bot]

Are you satisfied with the resolution of your issue?
Yes
No

[JerichoHy]

@hsm207 I tried to upgrade with pip install --upgrade tensorflow==2.0.0rc1, but problem existed stilly. Could you tell me how to upgrade to v2.0.0-rc0-101-gd2d2566eef 2.0.0-rc1. Thanks!

[hsm207]

@hsm207 I tried to upgrade with pip install --upgrade tensorflow==2.0.0rc1, but problem existed stilly. Could you tell me how to upgrade to v2.0.0-rc0-101-gd2d2566eef 2.0.0-rc1. Thanks!

@JerichoHy I did not do an upgrade. I created a new conda environment and ran pip install tensorflow==2.0.0-rc1

[JerichoHy]

@hsm207 I tried to upgrade with pip install --upgrade tensorflow==2.0.0rc1, but problem existed stilly. Could you tell me how to upgrade to v2.0.0-rc0-101-gd2d2566eef 2.0.0-rc1. Thanks!

@JerichoHy I did not do an upgrade. I created a new conda environment and ran pip install tensorflow==2.0.0-rc1

Thanks.

[Akoopie]

@JerichoHy I created a new conda environment with tensorflow==2.0.0-rc1 installed but still get the same error.

'OverflowError: Python int too large to convert to C long'
followed by traceback down to

'~\Miniconda3\lib\site-packages\tensorflow_core\python\ops\gen_sparse_ops.py in sparse_cross(indices, values, shapes, dense_inputs, hashed_output, num_buckets, hash_key, out_type, internal_type, name)
1147 shapes, dense_inputs, "hashed_output", hashed_output, "num_buckets",
1148 num_buckets, "hash_key", hash_key, "out_type", out_type,
-> 1149 "internal_type", internal_type)
1150 _result = _SparseCrossOutput._make(_result)
1151 return _result

SystemError: returned a result with an error set'

Happy to provide more info.

[tensorflow-bot]

Are you satisfied with the resolution of your issue?
Yes
No

[ymodak]

@Akoopie Please post a new issue describing your problem and provide all relevant information from the template. Thanks

[mberneti]

@ymodak The issue is OverflowError: Python int too large to convert to C long error after running this line of feature_columns.ipynb codes on the windows OS.
demo(feature_column.indicator_column(crossed_feature))

[d-gryphon]

This error is closed but it still happening in windows 10.

    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
OverflowError: Python int too large to convert to C long

[liqinglin54951]

`import tensorflow as tf #tf.version: '2.1.0'

from tensorflow import feature_column
from tensorflow.keras import layers

data = {'marks': [55,21,63,88,74,54,95,41,84,52],
'grade': ['average','poor','average','good','good','average','good','average','good','average'],
'point': ['c','f','c+','b+','b','c','a','d+','b+','c']}

def demo(feature_column):
feature_layer = layers.DenseFeatures(feature_column)
print(feature_layer(data).numpy())

marks = feature_column.numeric_column("marks")
marks_buckets = feature_column.bucketized_column(marks, boundaries=[30,40,50,60,70,80,90])

grade = feature_column.categorical_column_with_vocabulary_list(
'grade', ['poor', 'average', 'good']
)

crossed_feature = feature_column.crossed_column([marks_buckets, grade], hash_bucket_size=10)
demo(feature_column.indicator_column(crossed_feature))`

[liqinglin54951]

---

OverflowError Traceback (most recent call last)
OverflowError: Python int too large to convert to C long

The above exception was the direct cause of the following exception:

SystemError Traceback (most recent call last)
in
21
22 crossed_feature = feature_column.crossed_column([marks_buckets, grade], hash_bucket_size=10)
---> 23 demo(feature_column.indicator_column(crossed_feature))

in demo(feature_column)
11 def demo(feature_column):
12 feature_layer = layers.DenseFeatures(feature_column)
---> 13 print(feature_layer(data).numpy())
14
15 marks = feature_column.numeric_column("marks")

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py in call(self, inputs, *args, **kwargs)
820 with base_layer_utils.autocast_context_manager(
821 self._compute_dtype):
--> 822 outputs = self.call(cast_inputs, *args, **kwargs)
823 self._handle_activity_regularization(inputs, outputs)
824 self._set_mask_metadata(inputs, outputs, input_masks)

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\feature_column\dense_features.py in call(self, features, cols_to_output_tensors)
133 with ops.name_scope(column.name):
134 tensor = column.get_dense_tensor(transformation_cache,
--> 135 self._state_manager)
136 processed_tensors = self._process_dense_tensor(column, tensor)
137 if cols_to_output_tensors is not None:

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\feature_column\feature_column_v2.py in get_dense_tensor(self, transformation_cache, state_manager)
4349 # Feature has been already transformed. Return the intermediate
4350 # representation created by transform_feature.
-> 4351 return transformation_cache.get(self, state_manager)
4352
4353 @deprecation.deprecated(_FEATURE_COLUMN_DEPRECATION_DATE,

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\feature_column\feature_column_v2.py in get(self, key, state_manager)
2613 column = key
2614 logging.debug('Transforming feature_column %s.', column)
-> 2615 transformed = column.transform_feature(self, state_manager)
2616 if transformed is None:
2617 raise ValueError('Column {} is not supported.'.format(column.name))

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\feature_column\feature_column_v2.py in transform_feature(self, transformation_cache, state_manager)
4288 """
4289 id_weight_pair = self.categorical_column.get_sparse_tensors(
-> 4290 transformation_cache, state_manager)
4291 return self._transform_id_weight_pair(id_weight_pair)
4292

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\feature_column\feature_column_v2.py in get_sparse_tensors(self, transformation_cache, state_manager)
4140 """See CategoricalColumn base class."""
4141 return CategoricalColumn.IdWeightPair(
-> 4142 transformation_cache.get(self, state_manager), None)
4143
4144 @deprecation.deprecated(_FEATURE_COLUMN_DEPRECATION_DATE,

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\feature_column\feature_column_v2.py in get(self, key, state_manager)
2613 column = key
2614 logging.debug('Transforming feature_column %s.', column)
-> 2615 transformed = column.transform_feature(self, state_manager)
2616 if transformed is None:
2617 raise ValueError('Column {} is not supported.'.format(column.name))

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\feature_column\feature_column_v2.py in transform_feature(self, transformation_cache, state_manager)
4101 inputs=feature_tensors,
4102 num_buckets=self.hash_bucket_size,
-> 4103 hash_key=self.hash_key)
4104
4105 @deprecation.deprecated(_FEATURE_COLUMN_DEPRECATION_DATE,

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\ops\sparse_ops.py in sparse_cross_hashed(inputs, num_buckets, hash_key, name)
596 num_buckets=num_buckets,
597 hash_key=hash_key,
--> 598 name=name)
599
600

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\ops\sparse_ops.py in _sparse_cross_internal(inputs, hashed_output, num_buckets, hash_key, name)
649 out_type=out_type,
650 internal_type=internal_type,
--> 651 name=name)
652
653 return sparse_tensor.SparseTensor(indices_out, values_out, shape_out)

C:\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\ops\gen_sparse_ops.py in sparse_cross(indices, values, shapes, dense_inputs, hashed_output, num_buckets, hash_key, out_type, internal_type, name)
1046 "hashed_output", hashed_output, "num_buckets", num_buckets,
1047 "hash_key", hash_key, "out_type", out_type, "internal_type",
-> 1048 internal_type)
1049 _result = _SparseCrossOutput._make(_result)
1050 return _result

SystemError: built-in function TFE_Py_FastPathExecute returned a result with an error set

[liqinglin54951]

import tensorflow as tf # tf.version: 1.13.1 #run my code on https://www.katacoda.com/courses/tensorflow/playground
from tensorflow import feature_column

sess=tf.Session()#appended

data = {'marks': [55,21,63,88,74,54,95,41,84,52],
'grade': ['average','poor','average','good','good','average','good','average','good','average'],
'point': ['c','f','c+','b+','b','c','a','d+','b+','c']}

marks = feature_column.numeric_column("marks")
marks_buckets = feature_column.bucketized_column(marks, boundaries=[30,40,50,60,70,80,90])

grade = feature_column.categorical_column_with_vocabulary_list(
'grade', ['poor', 'average', 'good']
)

crossed_feature = feature_column.crossed_column([marks_buckets, grade], hash_bucket_size=10)

inputs = tf.feature_column.input_layer(data, [feature_column.indicator_column(crossed_feature)])
init = tf.global_variables_initializer()

init = tf.global_variables_initializer()
sess.run(tf.tables_initializer())
sess.run(init)
outputs=sess.run(inputs)
print(outputs)

I just changed my code for running on tensorflow version 1.13.1 from https://www.katacoda.com/courses/tensorflow/playground

[deathstroke111]

I'm getting the same error on Windows OS 10, tensorflow version 2.1.0 on gpu in conda environment using jupyter lab. final error message: <built-in function TFE_Py_FastPathExecute> returned a result with an error set . Is there any fix/workaround for it yet?