[On-Device-Training] Upgrade Flatbuffers to Support 2GB+ Checkpoints.

[AdamLouly]

Description

Modifications to support 2GB+ checkpoint & Upgrading Flatbuffers

Motivation and Context

This PR includes changes that will make ort handle 2GB+ checkpoints.
To do that we need to upgrade flatbuffers to 23.5.9 - google/flatbuffers#7945

• Modified the commitHash and the hash for the new version
• Removed the patch for rust generator's unused variable warning as it is no longer producing this - Check it out here
• Updated the VerifyField calls with alignment values that were introduced in the new version.

[edgchen1]

A few places that may need updating:

onnxruntime/onnxruntime/core/flatbuffers/schema/README.md

Lines 23 to 24 in a788514

	It is possible to use another flatc as well, e.g., from a separate installation. Note that ONNX Runtime uses
	FlatBuffers 1.12.

onnxruntime/cmake/external/onnxruntime_external_deps.cmake

Lines 101 to 108 in a788514

	#flatbuffers 1.11.0 does not have flatbuffers::IsOutRange, therefore we require 1.12.0+
	FetchContent_Declare(
	flatbuffers
	URL ${DEP_URL_flatbuffers}
	URL_HASH SHA1=${DEP_SHA1_flatbuffers}
	PATCH_COMMAND ${ONNXRUNTIME_FLATBUFFERS_PATCH_COMMAND}
	FIND_PACKAGE_ARGS 1.12.0...<2.0.0 NAMES Flatbuffers
	)

onnxruntime/js/web/package.json

Line 14 in a788514

"flatbuffers": "^1.12.0",

onnxruntime/js/web/package-lock.json

Line 12 in a788514

"flatbuffers": "^1.12.0",

@fs-eire would the onnxruntime-web flatbuffers dependency need to be updated as well?

[sumitsays]

From DirectML perspective, it looks good to me.

[fs-eire]

ONNX Runtime Web CI failed. This is because this PR does partially upgrade for flatbuffer for ort-web.

Please either revert this change for ort-web, or perform a full upgrade

• Use npm install flatbuffers@^24.3.7 to install so that packages-lock.json will be refreshed with dependencies version
• Follow instructions in js/web/onnxjs/ort-schema/flatbuffers/README.md to generate new flatbuffer files for v24.3.7 and update the files in js/web/onnxjs/ort-schema/flatbuffers/

We don't necessarily need this change for ort-web because only WebGL is using this flatbuffer library as dependencies, and we do not have plan to support on device training for WebGL backend. For Wasm/CPU/WebGPU, this library is not needed.

[AdamLouly]

ONNX Runtime Web CI failed. This is because this PR does partially upgrade for flatbuffer for ort-web.

Please either revert this change for ort-web, or perform a full upgrade

• Use npm install flatbuffers@^24.3.7 to install so that packages-lock.json will be refreshed with dependencies version
• Follow instructions in js/web/onnxjs/ort-schema/flatbuffers/README.md to generate new flatbuffer files for v24.3.7 and update the files in js/web/onnxjs/ort-schema/flatbuffers/

We don't necessarily need this change for ort-web because only WebGL is using this flatbuffer library as dependencies, and we do not have plan to support on device training for WebGL backend. For Wasm/CPU/WebGPU, this library is not needed.

Thanks for clarifying, in this case we can just revert ort-web upgrade.

[snnn]

LGTM. Thank you!

[snnn]

In file included from /mnt/vss/_work/1/s/include/onnxruntime/core/graph/graph.h:24, you need to add some guards to disable external warnings. Like what we do in https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/graph/onnx_protobuf.h . Otherwise the web pipelines won't pass

[pranavsharma]

Reviewing for admin.

[AdamLouly]

Thanks for the review everyone!

[snnn]

How is this file generated?