DirectRunner GameStats fails, ValueError: PCollection of size 2 with more than one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter'] #19540
Description
GameStats example is failing with following logs
*****************************************************
* Running GameStats with DirectRunner
*****************************************************
INFO:root:Missing
pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1137:
BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options
will not be supported
method_to_use = self._compute_method(p, p.options)
INFO:root:Running pipeline
with DirectRunner.
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:oauth2client.client:Refreshing
access_token
INFO:root:Created table google.com:clouddfe.goenka_python_validations.game_stats_teams
with schema <TableSchema
fields: [<TableFieldSchema
fields: []
mode: u'NULLABLE'
name: u'processing_time'
type:
u'STRING'>, <TableFieldSchema
fields: []
mode: u'NULLABLE'
name: u'total_score'
type: u'INTEGER'>,
<TableFieldSchema
fields: []
mode: u'NULLABLE'
name: u'window_start'
type: u'STRING'>, <TableFieldSchema
fields:
[]
mode: u'NULLABLE'
name: u'team'
type: u'STRING'>]>. Result: <Table
creationTime: 1559341252301
etag:
u'tEQeN0Sr4FvttBRZuXn7zw=='
id: u'google.com:clouddfe:goenka_python_validations.game_stats_teams'
kind:
u'bigquery#table'
lastModifiedTime: 1559341252336
location: u'US'
numBytes: 0
numLongTermBytes:
0
numRows: 0
schema: <TableSchema
fields: [<TableFieldSchema
fields: []
mode: u'NULLABLE'
name:
u'processing_time'
type: u'STRING'>, <TableFieldSchema
fields: []
mode: u'NULLABLE'
name: u'total_score'
type:
u'INTEGER'>, <TableFieldSchema
fields: []
mode: u'NULLABLE'
name: u'window_start'
type: u'STRING'>,
<TableFieldSchema
fields: []
mode: u'NULLABLE'
name: u'team'
type: u'STRING'>]>
selfLink: u'https://www.googleapis.com/bigquery/v2/projects/google.com:clouddfe/datasets/goenka_python_validations/tables/game_stats_teams'
tableReference:
<TableReference
datasetId: u'goenka_python_validations'
projectId: u'google.com:clouddfe'
tableId:
u'game_stats_teams'>
type: u'TABLE'>.
INFO:oauth2client.transport:Attempting refresh to obtain initial
access_token
INFO:oauth2client.client:Refreshing access_token
ERROR:root:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle
object at 0x7fb0f1cfb6d0>, due to an exception.
Traceback (most recent call last):
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 343, in call
finish_state)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 380, in attempt_call
evaluator.process_element(value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
line 633, in process_element
self.runner.process(element)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 753, in process
self._reraise_augmented(exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 807, in _reraise_augmented
raise_with_traceback(new_exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 751, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 564, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 571, in _invoke_process_per_window
side_inputs = [si[window] for si in self.side_inputs]
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
_FilteringIterable(self._iterable, target_window), self._view_options)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 346, in _from_runtime_iterable
return options['data'].view_fn(it)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 323, in <lambda>
lambda iterable: from_runtime_iterable(iterable, view_options))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 433, in _from_runtime_iterable
len(head), str(head[0]), str(head[1])))
ValueError: PCollection
of size 2 with more than one element accessed as a singleton view. First two elements encountered are
"181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
ERROR:root:Exception
at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception.
Traceback
(most recent call last):
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 343, in call
finish_state)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 380, in attempt_call
evaluator.process_element(value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
line 633, in process_element
self.runner.process(element)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 753, in process
self._reraise_augmented(exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 807, in _reraise_augmented
raise_with_traceback(new_exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 751, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 564, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 571, in _invoke_process_per_window
side_inputs = [si[window] for si in self.side_inputs]
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
_FilteringIterable(self._iterable, target_window), self._view_options)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 346, in _from_runtime_iterable
return options['data'].view_fn(it)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 323, in <lambda>
lambda iterable: from_runtime_iterable(iterable, view_options))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 433, in _from_runtime_iterable
len(head), str(head[0]), str(head[1])))
ValueError: PCollection
of size 2 with more than one element accessed as a singleton view. First two elements encountered are
"181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
ERROR:root:Exception
at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception.
Traceback
(most recent call last):
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 343, in call
finish_state)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 380, in attempt_call
evaluator.process_element(value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
line 633, in process_element
self.runner.process(element)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 753, in process
self._reraise_augmented(exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 807, in _reraise_augmented
raise_with_traceback(new_exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 751, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 564, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 571, in _invoke_process_per_window
side_inputs = [si[window] for si in self.side_inputs]
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
_FilteringIterable(self._iterable, target_window), self._view_options)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 346, in _from_runtime_iterable
return options['data'].view_fn(it)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 323, in <lambda>
lambda iterable: from_runtime_iterable(iterable, view_options))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 433, in _from_runtime_iterable
len(head), str(head[0]), str(head[1])))
ValueError: PCollection
of size 2 with more than one element accessed as a singleton view. First two elements encountered are
"181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
ERROR:root:Exception
at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7fb0f1cfb6d0>, due to an exception.
Traceback
(most recent call last):
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 343, in call
finish_state)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 380, in attempt_call
evaluator.process_element(value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
line 633, in process_element
self.runner.process(element)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 753, in process
self._reraise_augmented(exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 807, in _reraise_augmented
raise_with_traceback(new_exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 751, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 564, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 571, in _invoke_process_per_window
side_inputs = [si[window] for si in self.side_inputs]
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
_FilteringIterable(self._iterable, target_window), self._view_options)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 346, in _from_runtime_iterable
return options['data'].view_fn(it)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 323, in <lambda>
lambda iterable: from_runtime_iterable(iterable, view_options))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 433, in _from_runtime_iterable
len(head), str(head[0]), str(head[1])))
ValueError: PCollection
of size 2 with more than one element accessed as a singleton view. First two elements encountered are
"181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
ERROR:root:Giving
up after 4 attempts.
WARNING:root:A task failed with exception: PCollection of size 2 with more than
one element accessed as a singleton view. First two elements encountered are "181.611538462", "9.1".
[while running 'CalculateSpammyUsers/ProcessAndFilter']
INFO:oauth2client.transport:Attempting refresh
to obtain initial access_token
INFO:oauth2client.client:Refreshing access_token
Traceback (most recent
call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname,
loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py",
line 399, in <module>
run()
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/lib/python2.7/site-packages/apache_beam/examples/complete/game/game_stats.py",
line 393, in run
}, options.view_as(GoogleCloudOptions).project))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 426, in __exit__
self.run().wait_until_finish()
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
line 430, in wait_until_finish
self._executor.await_completion()
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 400, in await_completion
self._executor.await_completion()
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 446, in await_completion
raise_(t, v, tb)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 343, in call
finish_state)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py",
line 380, in attempt_call
evaluator.process_element(value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py",
line 633, in process_element
self.runner.process(element)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 753, in process
self._reraise_augmented(exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 807, in _reraise_augmented
raise_with_traceback(new_exn)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 751, in process
return self.do_fn_invoker.invoke_process(windowed_value)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 564, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
line 571, in _invoke_process_per_window
side_inputs = [si[window] for si in self.side_inputs]
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
line 65, in __getitem__
_FilteringIterable(self._iterable, target_window), self._view_options)
File
"/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 346, in _from_runtime_iterable
return options['data'].view_fn(it)
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 323, in <lambda>
lambda iterable: from_runtime_iterable(iterable, view_options))
File "/usr/local/google/home/goenka/rc_validations/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
line 433, in _from_runtime_iterable
len(head), str(head[0]), str(head[1])))
ValueError: PCollection
of size 2 with more than one element accessed as a singleton view. First two elements encountered are
"181.611538462", "9.1". [while running 'CalculateSpammyUsers/ProcessAndFilter']
Imported from Jira BEAM-7471. Original Jira may contain additional context.
Reported by: angoenka.