doesn't retry Google Cloud Storage "503 Backend Error"

[seamusabshere]

app: dbcrossbar
 ver: 0.4.2-beta.8
  Nov 17 20:11:19.349 DEBG Opt { log_format: Indented, log_extra: [], enable_unstable: false, cmd: Cp { command: Opt { if_exists: Overwrite, schema: Some(UnparsedLocator("dbcrossbar-schema:/tmp/schema20201117-16-12ceb9t.json")), temporaries: ["bigquery:myproject:tmp", "gs://mybucket", "s3://mybucket"], stream_size: None, from_args: [], to_args: [], where_clause: None, max_streams: 4, display_output_locators: false, from_locator: UnparsedLocator("s3://mybucket/mydir2/"), to_locator: UnparsedLocator("bigquery:myproject:tmp.mytable") } } }
  Nov 17 20:11:19.349 DEBG Configuration { path: "/root/.config/dbcrossbar/dbcrossbar.toml", doc: Document { root: Table(Table { items: {}, decor: Decor { prefix: "", suffix: "" }, implicit: false, position: Some(0) }), trailing: "" } }
  Nov 17 20:11:19.349 DEBG performing local data transfer
  from_locator: s3://mybucket/mydir2/
   Nov 17 20:11:19.349 DEBG getting CSV files from s3://mybucket/mydir2/
   Nov 17 20:11:19.349 DEBG listing s3://mybucket/mydir2/
  to_locator: bigquery:myproject:tmp.mytable
   to_temp: gs://mybucket/mytmp/
    Nov 17 20:11:19.349 DEBG deleting existing gs://mybucket/mytmp/
    Nov 17 20:11:19.349 DEBG listing gs://mybucket/mytmp/
  from_locator: s3://mybucket/mydir2/
   stream: my_chunk20201117-16-100pywq
    url: s3://mybucket/mydir2/myfile.csv
     Nov 17 20:11:19.824 DEBG streaming from s3://mybucket/mydir2/myfile.csv using `aws s3 cp`
   stream: my_chunk20201117-16-100w0xg
    url: s3://mybucket/mydir2/myfile.csv
     Nov 17 20:11:19.824 DEBG streaming from s3://mybucket/mydir2/myfile.csv using `aws s3 cp`
   stream: my_chunk20201117-16-101wb9q
    url: s3://mybucket/mydir2/myfile.csv
     Nov 17 20:11:19.834 DEBG streaming from s3://mybucket/mydir2/myfile.csv using `aws s3 cp`
   stream: my_chunk20201117-16-104ml8n
    url: s3://mybucket/mydir2/myfile.csv
     Nov 17 20:11:19.842 DEBG streaming from s3://mybucket/mydir2/myfile.csv using `aws s3 cp`
  to_locator: bigquery:myproject:tmp.mytable
   to_temp: gs://mybucket/mytmp/
    stream: my_chunk20201117-16-100pywq
     url: gs://mybucket/mytmp/myfile.csv
      Nov 17 20:11:19.846 DEBG streaming to gs://mybucket/mytmp/myfile.csv
    stream: my_chunk20201117-16-100w0xg
     url: gs://mybucket/mytmp/myfile.csv
      Nov 17 20:11:19.846 DEBG streaming to gs://mybucket/mytmp/myfile.csv
    stream: my_chunk20201117-16-101wb9q
     url: gs://mybucket/mytmp/myfile.csv
      Nov 17 20:11:19.846 DEBG streaming to gs://mybucket/mytmp/myfile.csv
    stream: my_chunk20201117-16-104ml8n
     url: gs://mybucket/mytmp/myfile.csv
      Nov 17 20:11:19.846 DEBG streaming to gs://mybucket/mytmp/myfile.csv
  from_locator: s3://mybucket/mydir2/
   stream: my_chunk20201117-16-105x6t0
    url: s3://mybucket/mydir2/myfile.csv
     Nov 17 20:11:22.729 DEBG streaming from s3://mybucket/mydir2/myfile.csv using `aws s3 cp`
  to_locator: bigquery:myproject:tmp.mytable
   to_temp: gs://mybucket/mytmp/
    stream: my_chunk20201117-16-105x6t0
     url: gs://mybucket/mytmp/myfile.csv
      Nov 17 20:11:22.730 DEBG streaming to gs://mybucket/mytmp/myfile.csv
  from_locator: s3://mybucket/mydir2/
   stream: my_chunk20201117-16-1064c3m
    url: s3://mybucket/mydir2/myfile.csv
     Nov 17 20:11:22.784 DEBG streaming from s3://mybucket/mydir2/myfile.csv using `aws s3 cp`
[... thousands of lines ...]
  from_locator: s3://mybucket/mydir2/
   stream: my_chunk20201117-16-1sxfp2x
    url: s3://mybucket/mydir2/myfile.csv
     Nov 17 20:26:08.632 DEBG streaming from s3://mybucket/mydir2/myfile.csv using `aws s3 cp`
  to_locator: bigquery:myproject:tmp.mytable
   to_temp: gs://mybucket/mytmp/
    stream: my_chunk20201117-16-1sxfp2x
     url: gs://mybucket/mytmp/myfile.csv
      Nov 17 20:26:08.638 DEBG streaming to gs://mybucket/mytmp/myfile.csv

[Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
download failed: s3://mybucket/mydir2/myfile.csv to - [Errno 32] Broken pipe
Error: POST error https://storage.googleapis.com/upload/storage/v1/b/mybucket/o?uploadType=media&ifGenerationMatch=0&name=mytmp%myfile.csvsv
  caused by: Google Cloud error: 503 Backend Error
download failed: s3://mybucket/mydir2/myfile.csv to - [Errno 32] Broken pipe
   0: failure::backtrace::internal::InternalBacktrace::new
   1: failure::backtrace::Backtrace::new
   2: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   3: dbcrossbarlib::drivers::gs::write_local_data::write_local_data_helper::{{closure}}::{{closure}}::{{closure}}
   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   5: futures_util::stream::stream::StreamExt::poll_next_unpin
   6: <S as futures_core::stream::TryStream>::try_poll_next
   7: <futures_util::stream::try_stream::try_collect::TryCollect<St,C> as core::future::future::Future>::poll
   8: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   9: dbcrossbar::cmd::cp::run::{{closure}}
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
  12: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  14: std::thread::local::LocalKey<T>::with
  15: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on
  16: tokio::runtime::context::enter
  17: tokio::runtime::handle::Handle::enter
  18: dbcrossbarlib::tokio_glue::run_futures_with_runtime
  19: dbcrossbar::main
  20: std::sys_common::backtrace::__rust_begin_short_backtrace
  21: std::rt::lang_start::{{closure}}
  22: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:373:40
      std::panicking::try
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:337:19
      std::panic::catch_unwind
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panic.rs:379:14
      std::rt::lang_start_internal
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/rt.rs:51:25
  23: main
download failed: s3://mybucket/mydir2/myfile.csv to - [Errno 32] Broken pipe

[emk]

If this is what I think it is, we've successfully made the initial connection to download the data, and the data has already started streaming through from source to destination. But then the download stream gets cut off mid-way, after some data has already been written to the destination.

There's no easy way to "retry" this, certainly not without replacing the current "Stream<Stream>" structure with something like a "Stream<() -> Stream>" structure.