[Import Model] High transient memory usage when installing large models with Docker

[davidkyle]

Using the docker container to run eland_import_hub_model and install a large model imposes large memory requirements. For example using the command below to install xlm-roberta-base requires the container to have more than 8GB of memory.

docker run -it --rm --network host \
    elastic/eland \
    eland_import_hub_model \
      --url 'https://elastic:XXX@host:9200/' \
      --hub-model-id xlm-roberta-base \
      --task-type fill_mask

If the container does not have enough memory the process exits shortly after the download has completed prior to uploading the model to Elasticsearch. Investigate the what is causing the high memory usage.

• What is causing the high memory usage?
  • Tracing the mode?
• How can a failure due to insufficient memory be detected and reported?