Logging experiments to S3 bucket

[qraleq]

I would like to have all the experiments stored to a S3 bucket, instead keeping them locally on the machine. If I'm not wrong, by default trains-server logs everything to /opt/trains/data/fileserver. Is it possible to connect trains-server directly to an S3 bucket without using an intermediate tool like s3fs or similar?

I'm able to upload the models and artifacts to an S3 bucket by using output_uri from trains, but I can't figure out how to log rest of the stuff (graphs, metrics, etc)

[bmartinn]

Hi @qraleq , as you mentioned, one obvious solution is to map /opt/trains/data/fileserver to your s3 bucket with s3fs.

The other maybe easier is simply to set the files_server to your s3 bucket.

Edit /trains.conf and replace this line
files_server: "http://localhost:8081"
With
files_server: "s3://mybucket/logs/"

And do not forget to add the AWS S3 credentials :)

BTW:
The console logs and the graphs themselves (not the debug images) are stored into the elastic search database in the trains-server

[qraleq]

@bmartinn Thank you for your answer! I'm interested in saving the logs and graphs to an S3 bucket, so I can access it from different machines. For example, I would run one instance for training, and I would like to have access to the experiments even when the training instance is offline using another instance which is available 24/7.

What's the best way to achieve this?

[bmartinn]

Yes @qraleq that is exactly why we designed the trains-server to be deployed as a server (as opposed to running it per user).

If your trains-server instance is running on the cloud, then you can do much more that view the logs, you can also share them, you can even remotely stop an experiment from the web UI (Abort), and not to forget the trains-agent addition, for creating your own cluster.
Anyway :)

Where are you running the current trains-server? You can quickly set up an instance on amazon with the trains-server AMI or with a simple docker-compose on any GCP / Azure cpu instance.

Once you have that, just configure the ~/trains.conf to point to the cloud trains-server, and you should be good to go.

What do you think?

[qraleq]

@bmartinn Thank you very much for your answer, I understand how you've structured things now!