Specifying `destination` in ProcessingOutput silently fails - and does not write to the destination S3 bucket

[cfregly]

This works:

sklearn_processor.run(code='preprocess-scikit.py',
                                       outputs=[ProcessingOutput(output_name='train_data', 
                                                                   source='/opt/ml/processing/output/train'),

This silently fails and does not write to the output s3 location:

s3_output_train_data = 's3://{}/{}/train'.format(bucket, output_prefix)

sklearn_processor.run(code='preprocess-scikit.py',
                                       outputs=[ProcessingOutput(output_name='train_data',
                                                                      source='/opt/ml/processing/output/train',
                                                                      destination=s3_output_train_data),

(Spent lots of time debugging this ^^)

Any suggestions?

[laurenyu]

@cfregly sorry for the delay in response here

a couple questions:

• if you do a DescribeProcessingJob call on one of the ones that silently failed, what is the output location? same as if you didn't specify one?
• when you run the job, are the output configs printed to the console? if so, does the destination look to be ignored there too?

[cfregly]

I can't seem to reproduce. Closing. I will re-open if this occurs again.