Transform fails when setting force_tf_compat_v1=False #3272

ConverJens · 2021-02-19T19:04:50Z

System information - Have I specified the code to reproduce the issue
(Yes/No): Yes

Environment in which the code is executed (e.g., Local (Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc): KubeFlow
TensorFlow version (you are using): 2.4.0
TFX Version: 0.27.0
Python version: 3.7

Describe the current behavior
Using to tf.strings.substr operations in the preprocessing fn used in Transform when setting force_tf_compat_v1=False fails while running in KubeFlow. Setting force_tf_compat_v1=True works.
Note: running in interactive mode, using force_tf_compat_v1=True causes python to crash and hence can't be tested properly.

Describe the expected behavior
Use native tf2 behaviour should work.

Standalone code to reproduce the issue Providing a bare minimum test case or
step(s) to reproduce the problem will greatly help us to debug the issue. If
possible, please share a link to Colab/Jupyter/any notebook.
Using attached file as preprocessing in KubeFlow with data generated by running (transform.py.zip):

data_path = "data.csv"
df = pd.DataFrame(data=[[random.randint(0,100), '2021-01-12T11:34:08'] for i in range(0, 100)], columns=["random_int", "datetime"])
df.head()
df.to_csv(os.path.join(data_root, data_path), index=False)

Other info / logs
This is the operation that I'm using in my preprocessing fn:

dt_str = tf.constant('2021-01-12T11:34:08')

year_str = tf.strings.substr(dt_str, pos=0, len=4, unit='UTF8_CHAR')
month_str = tf.strings.substr(dt_str, pos=5, len=2, unit='UTF8_CHAR')
day_str = tf.strings.substr(dt_str, pos=8, len=2, unit='UTF8_CHAR')
hour_str = tf.strings.substr(dt_str, pos=11, len=2, unit='UTF8_CHAR')
minute_str = tf.strings.substr(dt_str, pos=14, len=2, unit='UTF8_CHAR')
second_str = tf.strings.substr(dt_str, pos=17, len=2, unit='UTF8_CHAR')

Running locally this works in both eager and graph mode by setting tf.config.run_functions_eagerly(False/True).
However when running it through the Transform component and setting force_tf_compat_v1=False it fails with the following message:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in <module>
    main()
  File "/usr/local/lib/python3.7/dist-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main
    execution_info = launcher.launch()
  File "/usr/local/lib/python3.7/dist-packages/tfx/orchestration/launcher/base_component_launcher.py", line 209, in launch
    copy.deepcopy(execution_decision.exec_properties))
  File "/usr/local/lib/python3.7/dist-packages/tfx/orchestration/launcher/in_process_component_launcher.py", line 72, in _run_executor
    copy.deepcopy(input_dict), output_dict, copy.deepcopy(exec_properties))
  File "/pipeline/kubeflow/custom_components/transform_master/executor.py", line 493, in Do
    self.Transform(label_inputs, label_outputs, status_file)
  File "/pipeline/kubeflow/custom_components/transform_master/executor.py", line 1074, in Transform
    len(analyze_data_paths))
  File "/pipeline/kubeflow/custom_components/transform_master/executor.py", line 1209, in _RunBeamImpl
    preprocessing_fn, pipeline=pipeline))
  File "/usr/local/lib/python3.7/dist-packages/apache_beam/transforms/ptransform.py", line 1058, in __ror__
    return self.transform.__ror__(pvalueish, self.label)
  File "/usr/local/lib/python3.7/dist-packages/apache_beam/transforms/ptransform.py", line 573, in __ror__
    result = p.apply(self, pvalueish, label)
  File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 646, in apply
    return self.apply(transform, pvalueish)
  File "/usr/local/lib/python3.7/dist-packages/apache_beam/pipeline.py", line 689, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/runner.py", line 188, in apply
    return m(transform, input, options)
  File "/usr/local/lib/python3.7/dist-packages/apache_beam/runners/runner.py", line 218, in apply_PTransform
    return transform.expand(input)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_transform/beam/impl.py", line 1140, in expand
    self).expand(self._make_parent_dataset(dataset))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_transform/beam/impl.py", line 1087, in expand
    evaluate_schema_overrides=False)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_transform/schema_inference.py", line 196, in infer_feature_schema_v2
    metadata = collections.defaultdict(list, concrete_metadata_fn(inputs))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1669, in __call__
    return self._call_impl(args, kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1679, in _call_impl
    cancellation_manager)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1762, in _call_with_structured_signature
    cancellation_manager=cancellation_manager)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  pos 5 out of range for string at index 0
	 [[node Substr_1 (defined at pipeline/kubeflow/model/preprocessing/transform.py:96) ]] [Op:__inference_metadata_fn_1820]
Errors may have originated from an input operation.
Input Source operations connected to node Substr_1:
 inputs_copy (defined at usr/local/lib/python3.7/dist-packages/tensorflow_transform/tf_utils.py:81)
Function call stack:
metadata_fn

When setting force_tf_compat_v1=False it works as expected.

The text was updated successfully, but these errors were encountered:

zoyahav · 2021-02-22T11:40:38Z

Can you share with us statistics about this feature from StatisticsGen?
Specifically average/min/max length of the string in inputs["datetime"], and what percentage of it is missing in the dataset?

ConverJens · 2021-02-22T12:02:15Z

@zoyahav This is an extremely small test dataset, ~300 data points, so min=max=average length=19 and no items are missing, as is confirmed by output from StatisticsGen. As mentioned, this works perfectly when setting force_tf_compat_v1=True

ConverJens · 2021-02-24T10:14:46Z

@zoyahav This is the output from stats gen:

varshaan · 2021-02-25T19:02:42Z

Thanks! I am looking into this. Will provide an update when I have a fix.

ConverJens · 2021-02-25T20:37:27Z

@varshaan Thats great! Let me know if you need anything else from me.

ConverJens · 2021-03-22T13:35:11Z

@varshaan Any update on this?

varshaan · 2021-03-22T17:15:06Z

Hi, sorry I was away last week and not able to leave a comment here. I submitted a change to tf.Transform that I think should address this. Could you try using the latest TFX nightly from here?

ConverJens · 2021-03-23T14:16:39Z

@varshaan That's great! Our pipelines are still on TFX 0.27.0 and isn't fully TFX >= 0.28.0 compatible. I will try to bump TFX and try your fix as soon as I can and get back with the result.

ConverJens · 2021-04-01T09:41:28Z

@varshaan Now I've managed to test this and unfortunately the issue seems to remain for me.
If I try with the nightly release 20210318 I get the following (new) error:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/pyenv/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 360, in <module>
    main()
  File "/root/pyenv/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 353, in main
    execution_info = launcher.launch()
  File "/root/pyenv/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 209, in launch
    copy.deepcopy(execution_decision.exec_properties))
  File "/root/pyenv/lib/python3.7/site-packages/tfx/orchestration/launcher/in_process_component_launcher.py", line 72, in _run_executor
    copy.deepcopy(input_dict), output_dict, copy.deepcopy(exec_properties))
  File "/root/pyenv/lib/python3.7/site-packages/tfx/components/transform/executor.py", line 486, in Do
    self.Transform(label_inputs, label_outputs, status_file)
  File "/root/pyenv/lib/python3.7/site-packages/tfx/components/transform/executor.py", line 1009, in Transform
    len(analyze_data_paths))
  File "/root/pyenv/lib/python3.7/site-packages/tfx/components/transform/executor.py", line 1143, in _RunBeamImpl
    preprocessing_fn, pipeline=pipeline))
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/transforms/ptransform.py", line 1058, in __ror__
    return self.transform.__ror__(pvalueish, self.label)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/transforms/ptransform.py", line 573, in __ror__
    result = p.apply(self, pvalueish, label)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 646, in apply
    return self.apply(transform, pvalueish)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 689, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/runners/runner.py", line 188, in apply
    return m(transform, input, options)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/runners/runner.py", line 218, in apply_PTransform
    return transform.expand(input)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 1173, in expand
    self).expand(self._make_parent_dataset(dataset))
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 1120, in expand
    evaluate_schema_overrides=False)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow_transform/schema_inference.py", line 197, in infer_feature_schema_v2
    metadata = collections.defaultdict(list, optimized_concrete_fn())
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1669, in __call__
    return self._call_impl(args, kwargs)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py", line 247, in _call_impl
    args, kwargs, cancellation_manager)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1687, in _call_impl
    return self._call_with_flat_signature(args, kwargs, cancellation_manager)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1736, in _call_with_flat_signature
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  StringToNumberOp could not correctly convert string: 
	 [[node StringToNumber (defined at root/pyenv/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io_v2.py:430) ]] [Op:__inference_pruned_2112]
Function call stack:
pruned

And if I run the exact same code and data with 0.29.0rc0 I get an error that seems very similar to the one I first posted:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/pyenv/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 364, in <module>
    main()
  File "/root/pyenv/lib/python3.7/site-packages/tfx/orchestration/kubeflow/container_entrypoint.py", line 357, in main
    execution_info = launcher.launch()
  File "/root/pyenv/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 209, in launch
    copy.deepcopy(execution_decision.exec_properties))
  File "/root/pyenv/lib/python3.7/site-packages/tfx/orchestration/launcher/in_process_component_launcher.py", line 72, in _run_executor
    copy.deepcopy(input_dict), output_dict, copy.deepcopy(exec_properties))
  File "/root/pyenv/lib/python3.7/site-packages/tfx/components/transform/executor.py", line 466, in Do
    self.Transform(label_inputs, label_outputs, status_file)
  File "/root/pyenv/lib/python3.7/site-packages/tfx/components/transform/executor.py", line 985, in Transform
    len(analyze_data_paths))
  File "/root/pyenv/lib/python3.7/site-packages/tfx/components/transform/executor.py", line 1119, in _RunBeamImpl
    preprocessing_fn, pipeline=pipeline))
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/transforms/ptransform.py", line 1058, in __ror__
    return self.transform.__ror__(pvalueish, self.label)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/transforms/ptransform.py", line 573, in __ror__
    result = p.apply(self, pvalueish, label)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 646, in apply
    return self.apply(transform, pvalueish)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/pipeline.py", line 689, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/runners/runner.py", line 188, in apply
    return m(transform, input, options)
  File "/root/pyenv/lib/python3.7/site-packages/apache_beam/runners/runner.py", line 218, in apply_PTransform
    return transform.expand(input)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 1178, in expand
    self).expand(self._make_parent_dataset(dataset))
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow_transform/beam/impl.py", line 1125, in expand
    evaluate_schema_overrides=False)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow_transform/schema_inference.py", line 197, in infer_feature_schema_v2
    metadata = collections.defaultdict(list, optimized_concrete_fn())
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1669, in __call__
    return self._call_impl(args, kwargs)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/wrap_function.py", line 247, in _call_impl
    args, kwargs, cancellation_manager)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1687, in _call_impl
    return self._call_with_flat_signature(args, kwargs, cancellation_manager)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1736, in _call_with_flat_signature
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/root/pyenv/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  pos 5 out of range for string b'' at index 0
	 [[node Substr_1 (defined at root/pyenv/lib/python3.7/site-packages/tensorflow_transform/saved/saved_transform_io_v2.py:430) ]] [Op:__inference_pruned_2112]
Function call stack:
pruned

I would be very appreciative if you could have another look at this!

ConverJens · 2021-04-19T09:49:25Z

@varshaan Any update on this?

varshaan · 2021-04-19T18:16:07Z

Sorry, I missed your previous comment. Looking into this now.

ConverJens · 2021-04-20T10:08:44Z

No worries, thanks for investigating!

ConverJens · 2021-04-29T07:54:17Z

@varshaan Any update? Have you been able to reproduce?

varshaan · 2021-04-29T16:05:53Z

Yes, I was able to repro. I have a change in progress to address this. I will let you know once its submitted.

ConverJens · 2021-04-30T09:22:36Z

Great, thanks!

varshaan · 2021-05-13T19:30:04Z

Just leaving an update. I'm still working on fully testing this as the change is a bit non-trivial. I will try and get it submitted soon.

ConverJens · 2021-05-20T12:02:21Z

@varshaan Thanks for the update!

ConverJens · 2021-06-15T13:08:19Z

@varshaan Any update on this? Has your fix made it into the 1.0.0rc1 release?

axeltidemann · 2021-06-15T13:58:21Z

I have the exact same issue in the 1.0.0rc1 release, upgrading from 0.27.0.

varshaan · 2021-06-16T22:44:04Z

This commit is supposed to address this: tensorflow/transform@b878c66

It is not in the 1.0, but it should be in the next release. You can try testing with a nightly build post that commit if possible.

ConverJens · 2021-06-17T12:42:25Z

@varshaan Transform nightly version 1.1.0.dev20210617 works for me! Great job!

@axeltidemann Do you have the possibility to test this as well?

axeltidemann · 2021-06-18T09:12:10Z

Yes, this works for me using TFX 1.1.0.dev20210617.

varshaan · 2021-06-18T15:16:28Z

Thanks for verifying. I will close this issue.

google-ml-butler · 2021-06-18T17:34:12Z

Are you satisfied with the resolution of your issue?
Yes
No

ConverJens · 2021-11-03T08:45:13Z

I believe this issue has resurfaced for me in TFX 1.3.0.

@axeltidemann Is it the same for you?

@varshaan Has there been an regression in the latest TFX releases?

axeltidemann · 2021-11-03T09:01:35Z

@ConverJens I don't know for 1.3.0, I am on TFX 1.2.0 which does not have this issue.

varshaan · 2021-11-03T16:07:17Z

There should be no regression that I am aware of. The regression tests I added with the previous fix still pass. Could you provide a minimal repro so I can test it?

ConverJens · 2021-11-04T10:48:51Z

@varshaan Thanks for the quick response! I will test it a bit more and, if the issue persists, provide you with a minimal example.

ConverJens · 2021-11-24T08:43:41Z

@varshaan I've able to reproduce the issue now with TFX 1.4.0. It appears that the issue reappears when one sets disable_statistics=True in the Transform component. Otherwise it is exactly the same data and preprocessing logic as before.

I'm attaching a notebook (with data generation) which reproduces the issue locally for me.

transform_compat_tf1_issue.ipynb.zip

varshaan · 2021-11-29T18:08:01Z

Thanks for the repro! I'll take a look at this and get back to you by end of week.

varshaan · 2021-12-06T05:33:12Z

This issue reproduces as far back as TFX 1.2 if disable_statistics=True, so this isn't a regression. The original fix missed something in this code path. Thanks for reporting, I will send out a fix for this and update here after that.

ConverJens · 2021-12-08T09:17:40Z

@varshaan Great, thanks!

ConverJens · 2022-01-13T12:07:17Z

@varshaan Do you have an update for this issue?

varshaan · 2022-01-14T07:48:04Z

Hi, yes, I debugged this further and while the error message is the same, the source of the errors is different. It'll take me a couple more weeks to figure out exactly how to address what's happening in the disable_statistics=True case. Sorry about that, but I hope to have a fix by end of this month.

ConverJens · 2022-01-14T08:24:23Z

@varshaan No worries! Great work and thanks for the update!

varshaan · 2022-01-28T19:42:00Z

An update, I have a change in review to address this. Will comment back here once it is submitted.

ConverJens · 2022-01-31T12:37:21Z

Great! I'll happily test it as soon as a nightly build with the fix is out.

google-ml-butler · 2022-02-02T01:31:12Z

Are you satisfied with the resolution of your issue?
Yes
No

varshaan · 2022-02-02T01:59:56Z

Hi, the commit to address this was submitted earlier today [1]. Feel free to re-open this issue if you find that that didn't solve your problem.

[1] tensorflow/transform@718e394

ConverJens · 2022-04-11T11:54:27Z

@varshaan This issue now seems to be resolved. Thank you very much for your help!

ConverJens added the type:bug label Feb 19, 2021

arghyaganguly assigned arghyaganguly and nikelite and unassigned arghyaganguly Feb 22, 2021

arghyaganguly added the stat:awaiting tensorflower label Feb 22, 2021

dhruvesh09 mentioned this issue Mar 2, 2021

TFX 0.27.0 Issues #3193

Closed

arghyaganguly added stat:awaiting response and removed stat:awaiting tensorflower labels Mar 23, 2021

ConverJens mentioned this issue Mar 29, 2021

TFX nightly 0.29.0.dev20210328 create pipeline defs to large for KubeFlow #3459

Closed

arghyaganguly added stat:awaiting tensorflower and removed stat:awaiting response labels May 21, 2021

1025KB assigned varshaan Jun 18, 2021

1025KB closed this as completed Jun 18, 2021

varshaan reopened this Nov 29, 2021

copybara-service bot closed this as completed in tensorflow/transform@718e394 Feb 2, 2022

Transform fails when setting force_tf_compat_v1=False #3272

Transform fails when setting force_tf_compat_v1=False #3272

Comments

ConverJens commented Feb 19, 2021 • edited Loading

zoyahav commented Feb 22, 2021

ConverJens commented Feb 22, 2021

ConverJens commented Feb 24, 2021

varshaan commented Feb 25, 2021

ConverJens commented Feb 25, 2021

ConverJens commented Mar 22, 2021

varshaan commented Mar 22, 2021

ConverJens commented Mar 23, 2021

ConverJens commented Apr 1, 2021

ConverJens commented Apr 19, 2021

varshaan commented Apr 19, 2021

ConverJens commented Apr 20, 2021

ConverJens commented Apr 29, 2021

varshaan commented Apr 29, 2021

ConverJens commented Apr 30, 2021

varshaan commented May 13, 2021

ConverJens commented May 20, 2021

ConverJens commented Jun 15, 2021

axeltidemann commented Jun 15, 2021

varshaan commented Jun 16, 2021 • edited Loading

ConverJens commented Jun 17, 2021

axeltidemann commented Jun 18, 2021

varshaan commented Jun 18, 2021

google-ml-butler bot commented Jun 18, 2021

ConverJens commented Nov 3, 2021

axeltidemann commented Nov 3, 2021

varshaan commented Nov 3, 2021

ConverJens commented Nov 4, 2021

ConverJens commented Nov 24, 2021

varshaan commented Nov 29, 2021

varshaan commented Dec 6, 2021

ConverJens commented Dec 8, 2021

ConverJens commented Jan 13, 2022

varshaan commented Jan 14, 2022

ConverJens commented Jan 14, 2022

varshaan commented Jan 28, 2022

ConverJens commented Jan 31, 2022

google-ml-butler bot commented Feb 2, 2022

varshaan commented Feb 2, 2022

ConverJens commented Apr 11, 2022

ConverJens commented Feb 19, 2021 •

edited

Loading

varshaan commented Jun 16, 2021 •

edited

Loading