[Bug]: GCS IO publishes uninformative error message

[tvalentyn]

What happened?

When I ran a wordcount pipeline with an output set to a bucket in a different project (causing permission errors), I saw following output in the console:

  File "apache_beam/runners/common.py", line 1731, in apache_beam.runners.common._OutputHandler.finish_bundle_outputs
  File "/usr/local/lib/python3.9/site-packages/apache_beam/io/iobase.py", line 1202, in finish_bundle
    self.writer.close(),
  File "/usr/local/lib/python3.9/site-packages/apache_beam/io/filebasedsink.py", line 434, in close
    self.sink.close(self.temp_handle)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/io/textio.py", line 524, in close
    super().close(file_handle)
  File "/usr/local/lib/python3.9/site-packages/apache_beam/io/filebasedsink.py", line 167, in close
    file_handle.close()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/io/filesystemio.py", line 215, in close
    self._uploader.finish()
  File "/usr/local/lib/python3.9/site-packages/apache_beam/io/gcp/gcsio.py", line 829, in finish
    self._upload_thread.last_error.message)  # pylint: disable=raising-bad-type
AttributeError: 'HttpForbiddenError' object has no attribute 'message' [while running 'Write/Write/WriteImpl/WriteBundles-ptransform-58']

Looks like not all possible errors have the message attribute, and the code needs to account for it.

Relevant PR: #24449

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tvalentyn]

cc: @BjornPrime who is working on GCS IO

[BjornPrime]

The client swap should take care of this. I'll claim it for now and close it once the swap is merged.

[BjornPrime]

.take-issue

[cozos]

This code:

raise type(self._upload_thread.last_error)(
          "Error while uploading file %s: %s",
          self._path,
          self._upload_thread.last_error.message if hasattr(self._upload_thread.last_error, "message") else "")  # pylint: disable=raising-bad-type

Totally breaks Google's HttpError for me, since it overrides response which is expected to be dictionary rather than a message.

Calling HttpError.status_code results in this exception:

File "/generate_dataset_docker_pybinary.runfiles/cruise_ws/cruise/mlp/cfs/projects/temporal_understanding/generate_dataset_docker_pybinary_exedir/apitools/base/py/exceptions.py", line 78, in status_code
    return int(self.response['status'])
TypeError: string indices must be integers [while running 'Publish dataset/Write to big query/StoreArtifacts/WriteArtifact-ptransform-158']

[cozos]

This should probably be:

raise BeamGcsError("error while uploading file %s...") from self._upload_thread.last_error

[cozos]

@tvalentyn If you agree I can make a PR

[liferoad]

Note that #28079 will remove apitools. @BjornPrime FYI.

[tvalentyn]

@cozos SGTM. You can raise a RuntimeError. As mentioned, this would hopefully be moot soon though.

[cozos]

@tvalentyn #28470

[tvalentyn]

Thanks, @cozos !