Late bind external data

[pankajroark]

Instead of bundling external data into the docker image, download it at model load time using b10cp. b10cp download is very fast and we avoid the decompression step that comes with bundling into docker image. We also avoid the longer build times by bundling weights into docker image.

With https://github.com/basetenlabs/flan-t5-xl-truss/tree/ext-data Downloading the weights, which total around 11 GB, takes 40 seconds compared to 2 minutes when downloading from HuggingFace directly. This is significant improvement in startup time.

[bolasim]

lgtm pending tests passing

[bolasim]

One thing that's not clear to me here (maybe I missed it in the last PR),

Where do we ignore that paths in external_data when zipping up the PR? I worry that users will end up with a copy in their image anyways if they don't clean up the data_dir before deploying their model (and I know what the client doesn't check for external_data at this point)

[pankajroark]

One thing that's not clear to me here (maybe I missed it in the last PR),

Where do we ignore that paths in external_data when zipping up the PR? I worry that users will end up with a copy in their image anyways if they don't clean up the data_dir before deploying their model (and I know what the client doesn't check for external_data at this point)

You're right, at this time this will need to be taken care of manually. In future, we should provide a truss cli feature similar to git lfs track, where we do this automatically, i.e. upload file to provided storage, remove local file and update external_data.