Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Fixing directory issue of local metadataprovider with aws batch. #141
There is an issue in syncing metadata for local metadata provider when using AWS Batch.
I am using a local metadata provider with executes on Batch with S3 datastore. I was able to complete the flow successfully on batch and was trying to analyze results in a notebook. But when I access properties of the run like
I am using https://github.com/valayDave/mnist-experiments-with-metaflow/blob/without_conda_test/hello_mnist.py as the file and running the following command :
The odd thing is that there are two
I investigated that there is data sync after a step from S3 which brings metadata back to the client and does a copy tree operation.
While on batch because of the METAFLOW_DATASTORE_SYSROOT_LOCAL is being referenced to DATASTORE_LOCAL_DIR which is
Removing the METAFLOW_DATASTORE_SYSROOT_LOCAL the fixed the problem.