Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

InvalidArgumentError: indices[0,4] = 10 is not in [0, 10) [[node model/embedding_2/embedding_lookup (defined at <ipython-input-54-57c88bb5c904>:38) ]] [Op:__inference_distributed_function_20003] Errors may have originated from an input operation. Input Source operations connected to node model/embedding_2/embedding_lookup: model/embedding_2/embedding_lookup/19444 (defined at c:\users\naik9\appdata\local\programs\python\python37\lib\contextlib.py:112) Function call stack: distributed_function #37254

Closed
dineshnaikb opened this issue Mar 3, 2020 · 6 comments
Assignees
Labels
comp:keras Keras related issues type:bug Bug

Comments

@dineshnaikb
Copy link

Train on 10240 samples, validate on 1284 samples
Epoch 1/30
40/10240 [..............................] - ETA: 4:26WARNING:tensorflow:Can save best model only with val_categorical_accuracy available, skipping.

InvalidArgumentError Traceback (most recent call last)
in
----> 1 train(model_cnn,'cnn', use_pos, use_meta, use_dep)

in train(model, name, use_pos, use_meta, use_dep)
36 {'main_input': X_val, 'aux_input': X_val_meta, 'dep_input': X_val_dep},
37 {'main_output': Y_val}
---> 38 ), callbacks=[tb,csv_logger,checkpoint])
39 else:
40 model.fit(

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
817 max_queue_size=max_queue_size,
818 workers=workers,
--> 819 use_multiprocessing=use_multiprocessing)
820
821 def evaluate(self,

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
340 mode=ModeKeys.TRAIN,
341 training_context=training_context,
--> 342 total_epochs=epochs)
343 cbks.make_logs(model, epoch_logs, training_result, ModeKeys.TRAIN)
344

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in run_one_epoch(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)
126 step=step, mode=mode, size=current_batch_size) as batch_logs:
127 try:
--> 128 batch_outs = execution_function(iterator)
129 except (StopIteration, errors.OutOfRangeError):
130 # TODO(kaftan): File bug about tf function and errors.OutOfRangeError?

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2_utils.py in execution_function(input_fn)
96 # numpy translates Tensors to values in Eager mode.
97 return nest.map_structure(_non_none_constant_value,
---> 98 distributed_function(input_fn))
99
100 return execution_function

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\eager\def_function.py in call(self, *args, **kwds)
566 xla_context.Exit()
567 else:
--> 568 result = self._call(*args, **kwds)
569
570 if tracing_count == self._get_tracing_count():

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\eager\def_function.py in _call(self, *args, **kwds)
630 # Lifting succeeded, so variables are initialized and we can run the
631 # stateless function.
--> 632 return self._stateless_fn(*args, **kwds)
633 else:
634 canon_args, canon_kwds = \

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\eager\function.py in call(self, *args, **kwargs)
2361 with self._lock:
2362 graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 2363 return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
2364
2365 @Property

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\eager\function.py in _filtered_call(self, args, kwargs)
1609 if isinstance(t, (ops.Tensor,
1610 resource_variable_ops.BaseResourceVariable))),
-> 1611 self.captured_inputs)
1612
1613 def _call_flat(self, args, captured_inputs, cancellation_manager=None):

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1690 # No tape is watching; skip to running the function.
1691 return self._build_call_outputs(self._inference_function.call(
-> 1692 ctx, args, cancellation_manager=cancellation_manager))
1693 forward_backward = self._select_forward_and_backward_functions(
1694 args,

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\eager\function.py in call(self, ctx, args, cancellation_manager)
543 inputs=args,
544 attrs=("executor_type", executor_type, "config_proto", config),
--> 545 ctx=ctx)
546 else:
547 outputs = execute.execute_with_cancellation(

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\tensorflow_core\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
65 else:
66 message = e.message
---> 67 six.raise_from(core._status_to_exception(e.code, message), None)
68 except TypeError as e:
69 keras_symbolic_tensors = [

c:\users\n\appdata\local\programs\python\python37\lib\site-packages\six.py in raise_from(value, from_value)

InvalidArgumentError: indices[0,4] = 10 is not in [0, 10)
[[node model/embedding_2/embedding_lookup (defined at :38) ]] [Op:__inference_distributed_function_20003]

Errors may have originated from an input operation.
Input Source operations connected to node model/embedding_2/embedding_lookup:
model/embedding_2/embedding_lookup/19444 (defined at c:\users\n\appdata\local\programs\python\python37\lib\contextlib.py:112)

Function call stack:
distributed_function

@dineshnaikb
Copy link
Author

@Saduf2019 Saduf2019 assigned Saduf2019 and unassigned amahendrakar Mar 3, 2020
@Saduf2019
Copy link
Contributor

Saduf2019 commented Mar 3, 2020

@Hackerdash
please refer to this link, let us know if it helps.

please provide with simple standalone code for us to replicate the issue in our environment with correct indentation and all dependencies along with the tensorflow version to replicate in.Thanks!

@lucifer2288
Copy link

same issue

@Saduf2019
Copy link
Contributor

Could you please update us on the above comment on standalone code to help us resolve your issue

@Saduf2019 Saduf2019 added stat:awaiting response Status - Awaiting response from author comp:keras Keras related issues and removed stat:awaiting response Status - Awaiting response from author labels Mar 11, 2020
@Saduf2019
Copy link
Contributor

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

@tensorflow-bot
Copy link

Are you satisfied with the resolution of your issue?
Yes
No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:keras Keras related issues type:bug Bug
Projects
None yet
Development

No branches or pull requests

4 participants