MANTA-5342: mako_rollup should survive reprovisioning

[bahamat]

See also TritonDataCenter/mako-gc-cmon-plugins#1

Additionally, zones should be hot patched with the following

manta-oneach -s storage '
  mkdir -p /manta/mako_rollup
  rsync -a /var/tmp/mako_rollup/ /manta/mako_rollup/
  rm -rf /var/tmp/mako_rollup
  ln -s ../../manta/mako_rollup/ /var/tmp/mako_rollup
'

[askfongjojo]

The unwrap function in rust doesn't understand the symbolic link:

# before the hotpatch
[root@505679d6 (storage) ~]$ ls -l /var/tmp/mako_rollup/
total 2
-rw-r--r-- 1 root root 827 Jul 10 08:01 mako_rollup.out

# apply the hotpatch
[root@505679d6 (storage) ~]$ mkdir -p /manta/mako_rollup
[root@505679d6 (storage) ~]$ rsync -a /var/tmp/mako_rollup/ /manta/mako_rollup/
[root@505679d6 (storage) ~]$ rm -rf /var/tmp/mako_rollup
[root@505679d6 (storage) ~]$ ln -s ../var/tmp/mako_rollup /manta/mako_rollup
[root@505679d6 (storage) ~]$ ls -l /manta/mako_rollup
total 2
lrwxrwxrwx 1 root root  22 Jul 11 00:23 mako_rollup -> ../var/tmp/mako_rollup
-rw-r--r-- 1 root root 827 Jul 10 08:01 mako_rollup.out

# run the cron job as is (this is for the scenario of a storage zone not having the fix)
[root@505679d6 (storage) ~]$ RUST_BACKTRACE=1 /opt/smartdc/mako/bin/upload_mako_ls.sh >>/var/log/mako-ls-upload.log     
Sat Jul 11 00:30:03 UTC 2020: upload_mako_ls.sh: info: starting mako rollup
thread 'main' panicked at 'metadata call failed: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:997:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::continue_panic_fmt
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::result::unwrap_failed
   9: mako_rollup::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
  12: __rust_maybe_catch_panic
  13: std::rt::lang_start_internal
  14: main
  15: _start_crt

We probably have to go without the hotpatch, i.e. we have to back up the existing mako_rollup.out file more time until we upgrade to a version that has the fix.

[bahamat]

Does it leave a core file?

[askfongjojo]

Does it leave a core file?

No core file (since this is bash calling rust). You can access my sandbox to debug further.

[bahamat]

I see what happened. For the hot patch instructions I got the symlink backward which created a dangling symlink.

Should be

ln -s ../../manta/mako_rollup/ /var/tmp/mako_rollup

not

ln -s ../var/tmp/mako_rollup /manta/mako_rollup

Fixed in my comment at top.