[BE] add SHFMT linter to format shell scripts

[XuehaiPan]

Stack from ghstack (oldest at bottom):

• [BE][4/4] format shell scripts with SHFMT in scripts/ #157689
• [BE][3/4] format shell scripts with SHFMT in .ci/ #157688
• [BE][2/4] format shell scripts with SHFMT in .circleci/ and .github/ #157687
• [BE][1/4] format shell scripts with SHFMT #157686
• -> [BE] add SHFMT linter to format shell scripts #157685

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @jerryzh168 @seemethere @malfet @pytorch/pytorch-dev-infra @albanD @voznesenskym @penguinwu @EikanWang @Guobing-Chen @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @ZainRizvi @huydhn @clee2000

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157685

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 2 Unrelated Failures

As of commit cebe69d with merge base 9aedb3c ():

NEW FAILURES - The following jobs have failed:

• linux-s390x-binary-manywheel / manywheel-py3_10-cpu-s390x-build / build (gh)
  undefined reference to 'dlclose'
• linux-s390x-binary-manywheel / manywheel-py3_11-cpu-s390x-build / build (gh)
  undefined reference to 'dlclose'
• linux-s390x-binary-manywheel / manywheel-py3_12-cpu-s390x-build / build (gh)
  undefined reference to 'dlclose'
• linux-s390x-binary-manywheel / manywheel-py3_13-cpu-s390x-build / build (gh)
  undefined reference to 'dlclose'
• linux-s390x-binary-manywheel / manywheel-py3_13t-cpu-s390x-build / build (gh)
  undefined reference to 'dlclose'
• linux-s390x-binary-manywheel / manywheel-py3_14-cpu-s390x-build / build (gh)
  undefined reference to 'dlclose'
• linux-s390x-binary-manywheel / manywheel-py3_14t-cpu-s390x-build / build (gh)
  undefined reference to 'dlclose'

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 3, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (similar failure)
  test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_flex_attention_noncontig_with_holes_False_cross_attention_False_cuda_float32

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 5, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
  test_nestedtensor.py::TestNestedTensorSubclassCUDA::test_flex_attention_converts_stacked_seq_indices_cuda_float32

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Skylion007]

Do we have shellcheck enabled? If not, may be a less intrusive check to start with.

[XuehaiPan]

Do we have shellcheck enabled?

No.

[seemethere]

Do we have shellcheck enabled?

No.

I actually do believe we have shellcheck enabled, it's just over a smaller subset of things:

pytorch/.lintrunner.toml

Lines 886 to 905 in 86670b3

	[[linter]]
	code = 'SHELLCHECK'
	include_patterns = [
	'.ci/pytorch/**/*.sh'
	]
	exclude_patterns = [
	'**/fb/**',
	]
	command = [
	'python3',
	'tools/linter/adapters/shellcheck_linter.py',
	'--',
	'@{{PATHSFILE}}',
	]
	init_command = [
	'python3',
	'tools/linter/adapters/pip_init.py',
	'--dry-run={{DRYRUN}}',
	'shellcheck-py==0.7.2.1',
	]

What's the benefit here of shfmt over just enabling shellcheck everywhere else?

[XuehaiPan]

I actually do believe we have shellcheck enabled, it's just over a smaller subset of things:

Thanks for the clarification.

What's the benefit here of shfmt over just enabling shellcheck everywhere else?

It's a formatter to standardize the style for shell scripts and prevent potential merge conflicts like PYFMT for Python code.

FYI, shellcheck --format=diff provides an autofix feature.