[Test][CI][gfx1030][test_soft_max] Investigate & fix soft_max failures and instabilities

[atamazov]

PR #240 has a workaround, commit b4d6805 [test_soft_max] Workaround for cases that often fail on Jenkins and always fail with ROCm3.3/Radeon VII.

Nevertheless, test_soft_max continues to often fail on jenkins, for example, see:

• http://micimaster.amd.com/blue/organizations/jenkins/MLLibs%2FMIOpen/detail/develop/17/pipeline/
• http://micimaster.amd.com/blue/organizations/jenkins/MLLibs%2FMIOpen/detail/sqlite-binary-cache-reenable/3/pipeline/952

[aserio]

@daniellowell what is the status of this ticket?

[aserio]

@daniellowell following up with my request.

[atamazov]

This still happens from time to time.

[junliume]

This still happens from time to time, more often on gfx1030 servers:
http://micimaster.amd.com/blue/organizations/jenkins/MLLibs%2FMIOpen/detail/composable_kernel_init_integration_v3/35/pipeline
CC: @asroy @aserio @atamazov