Multiple jobs output_data configurations doesn't work independently

[maroony]

Problem Description

If you use multiple output_data configurations in your jobs.yaml file, they do not work independently. If the first will end in an error, the second won't start. This is important, when you configure one for log files.

Batch Shipyard Version

3.8.1

[alfpark]

Can you please provide an example configuration that demonstrates this?

edit: I'm assuming you have a sequence of azure_storage items, and that if one of those fails, then they all fail. Is this correct?

[alfpark]

This is limited only to non-native pools.

[maroony]

This is a snippet from my jobs.yaml file:

   output_data:
      azure_storage:
        # data
      - storage_account_settings: my_storage_account
        local_path: ${AZ_BATCH_TASK_DIR}/data
        remote_path: /my/remote/path/to/data
        is_file_share: false
        condition: taskcompletion  
        # logs
      - storage_account_settings: my_storage_account
        local_path:  ${AZ_BATCH_TASK_DIR}/logs"
        remote_path: /my/remote/path/to/logs
        is_file_share: false
        condition: taskcompletion

Here you can see 2 output_data configurations. If there is an error within the data transfer, the transfer for the logs won't start. So my workaround for the moment is to switch these configurations, to always get my logs.

[alfpark]

Thanks for the additional details, this has been fixed in develop. Please provide feedback using the develop branch or the develop-cli Docker image - you will need to recreate your pool.