S3 related configuration

[petrovicboban]

Can someone provide me with service, executor, and s3uploder configs, so I could see S3logs of my tasks in UI?
Thanks in advance

[petrovicboban]

Is there any way I can get help here?
We have discovered that executor creates new uploader:

INFO  [2018-03-23 09:23:29,715] [main] c.h.s.s3uploader.SingularityS3UploaderDriver 8765 - Created new uploader SingularityS3Uploader [uploadMetadata=S3UploadMetadata [directory=/var/mesos/slaves/4b0e9fed-0308-42ba-b856-d53e81fdb92e-S13/frameworks/Singularity/executors/tao5/runs/2acdee44-b2bc-4b85-8565-addb8ac48795/boban-test-32-1521797006456-1-db05-DEFAULT, fileGlob=service.log*.[gb]z*, s3Bucket=singularity, s3KeyFormat=boban-test/%Y/%m/boban-test-32-1521797006456-1-db05-DEFAULT_%index-%s%fileext, finished=false, onFinishGlob=Optional.absent(), pid=Optional.absent(), s3AccessKey=Optional.absent(), s3SecretKey=Optional.absent(), finishedAfterMillisWithoutNewFile=Optional.absent(), s3StorageClass=Optional.of(STANDARD_IA), applyStorageClassIfOverBytes=Optional.of(75000), uploadImmediately=Optional.of(false)], metadataPath=/var/log/singularity/s3/metadata/boban-test-32-1521797006456-1-db05-DEFAULT-file1.s3.json]

Since executor compress service.log and saves it to SANDBOX/log folder, uploader can't find it. I tried adding :

s3UploaderAdditionalFiles:
   - directory: logs

to /etc/singularity.executor.yaml, /etc/singularity.executor.cleanup.yaml, /etc/singularity.s3base.yaml but nothing has changed.

Version 0.13 has SANDBOX/logs folder in search path by default, so it works when using that version.

[ssalinas]

Did you read through the release notes for 0.14 already? There is a section on configuration updates specifically related to s3 uploader configs https://github.com/HubSpot/Singularity/releases/tag/Singularity-0.14.0

[petrovicboban]

I did it many times, tried everything. No success.

  s3UploaderAdditionalFiles:
   - filename: access.log
     # The default directory in the executor was set to 'logs', now it must be manually specified
     # If not specified, the directory to search for log files will be the task app directory in the sandbox
     directory: logs

I need to upload file called service.log-201803231521797019.log.gz in logs folder.
Can I use fileGlob in s3UploaderAdditionalFiles ?

[ssalinas]

I'll edit the release notes later to make it more clear. In the notes it states that that config you posted gets removed from executor, but needs to be added to singularity service yaml. From the current release notes.

# in SingularityService yaml configuration
s3:
  s3Bucket: my-logs-bucket
  s3KeyFormat: "%requestId/%Y/%m/%taskId_%index-%s-%filename"
  s3StorageClass: "STANDARD_IA"
  applyS3StorageClassAfterBytes: 75000
  s3UploaderAdditionalFiles:
   - filename: access.log
     # The default directory in the executor was set to 'logs', now it must be manually specified
     # If not specified, the directory to search for log files will be the task app directory in the sandbox
     directory: logs

(note the two storage class related configs are optional)

[ssalinas]

I've updated the release notes to make this more obvious. All you should need is something like:

s3UploaderAdditionalFiles:
  - filename: service.log
    directory: logs

in the service yaml config

[petrovicboban]

That's good, now it works fine! Thanks for this.
Is there something I should do in order for UI to show uploaded log files to s3? If I remember correctly, UI was doing that some time before.

[ssalinas]

That part should have remained similar. If you had a custom s3KeyFormat in your old executor config, you'll want to move that to SingularityService as well, since it uses that to search for logs in s3

[petrovicboban]

s3KeyFormat had and has default value now. It's interesting that I don't see logs that have been just updated.

[petrovicboban]

How can I debug it?

[petrovicboban]

Missing S3 logs from UI is related to #1640. So it is fixed #1762