Image uploader script

[jprochazk]

Closes #2132

What

Adds just upload <image>, which uploads the image to google cloud using gsutil. The resulting URL contains the content hash plus the original file name, which ensures that it is unique.

$ just upload image.png
https://static.rerun.io/dd31922030a6bf7223d5e3728d8da5407f4d6b1a_image.png

The script itself contains instructions for how to setup gsutil. It's really just a link to Google's tutorial. Eventually it will be documented separately together with the rest of the new workflow.

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)

PR Build Summary: https://build.rerun.io/pr/2164

[jprochazk]

Example image uploaded via this script: https://static.rerun.io/dd31922030a6bf7223d5e3728d8da5407f4d6b1a_minimal.png

[emilk]

I installed it, and this is my reward:

❯  gsutil --help
ImportError: dlopen(/Users/emilk/.pyenv/versions/3.8.12/lib/python3.8/site-packages/_cffi_backend.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_ffi_prep_closure'
thread '<unnamed>' panicked at 'Python API call failed', /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.15.2/src/err/mod.rs:582:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/emilk/Downloads/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 16, in <module>
    import bootstrapping
  File "/Users/emilk/Downloads/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 50, in <module>
    from googlecloudsdk.core.credentials import store as c_store
  File "/Users/emilk/Downloads/google-cloud-sdk/lib/googlecloudsdk/core/credentials/store.py", line 34, in <module>
    from google.auth import external_account as google_auth_external_account
  File "/Users/emilk/Downloads/google-cloud-sdk/lib/third_party/google/auth/external_account.py", line 42, in <module>
    from google.auth import impersonated_credentials
  File "/Users/emilk/Downloads/google-cloud-sdk/lib/third_party/google/auth/impersonated_credentials.py", line 39, in <module>
    from google.auth import jwt
  File "/Users/emilk/Downloads/google-cloud-sdk/lib/third_party/google/auth/jwt.py", line 57, in <module>
    from google.auth import _service_account_info
  File "/Users/emilk/Downloads/google-cloud-sdk/lib/third_party/google/auth/_service_account_info.py", line 22, in <module>
    from google.auth import crypt
  File "/Users/emilk/Downloads/google-cloud-sdk/lib/third_party/google/auth/crypt/__init__.py", line 43, in <module>
    from google.auth.crypt import rsa
  File "/Users/emilk/Downloads/google-cloud-sdk/lib/third_party/google/auth/crypt/rsa.py", line 20, in <module>
    from google.auth.crypt import _cryptography_rsa
  File "/Users/emilk/Downloads/google-cloud-sdk/lib/third_party/google/auth/crypt/_cryptography_rsa.py", line 25, in <module>
    from cryptography.hazmat.primitives import serialization
  File "/Users/emilk/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cryptography/hazmat/primitives/serialization/__init__.py", line 16, in <module>
    from cryptography.hazmat.primitives.serialization.base import (
  File "/Users/emilk/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cryptography/hazmat/primitives/serialization/base.py", line 9, in <module>
    from cryptography.hazmat.primitives.asymmetric.types import (
  File "/Users/emilk/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/types.py", line 8, in <module>
    from cryptography.hazmat.primitives.asymmetric import (
  File "/Users/emilk/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/dsa.py", line 10, in <module>
    from cryptography.hazmat.primitives.asymmetric import utils as asym_utils
  File "/Users/emilk/.pyenv/versions/3.8.12/lib/python3.8/site-packages/cryptography/hazmat/primitives/asymmetric/utils.py", line 6, in <module>
    from cryptography.hazmat.bindings._rust import asn1
pyo3_runtime.PanicException: Python API call failed

god damn you python.

gcloud --help works though

[jprochazk]

I changed the script to use the gcloud client library. If gcloud works, hopefully gcloud init should too, which should be enough to use this. Strange that the Rust bindings out of all things failed to install 🤔.

[emilk]

Sadly I still get the same error 😭

[jprochazk]

How did you install it? I remember using apt (https://cloud.google.com/storage/docs/gsutil_install#deb)

[emilk]

I followed their instructions and downloaded https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-432.0.0-darwin-x86_64.tar.gz

[jprochazk]

You could try installing via brew install --cask google-cloud-sdk, I tried it on my mac and it seems to work

[emilk]

I tried the brew version now; same problem. I'll take a deeper look after the 0.6 release

[emilk]

It would be nice to run pngcrush -rem allb -brute -reduce on all pngs, but I guess that is something we can add later (and perhaps even do in batch on everything uploaded)

[emilk]

Looks like if I do scripts/upload_image.py /Users/emilk/image.pg, that will result in a really bad object_name. I think we should just use the filename here, not the whole path

[Wumpf]

works like a charm for me :)