Update MaxEngine to optionally return token log probability

[wenxindongwork]

Description

This PR and AI-Hypercomputer/JetStream#260 together enables MaxEngine to return token log probability when return_log_prob is set to True in the global config.

This change is needed for calculating GRPO loss.

Tests

Manually tested

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.

[bvandermoon]

Do you know why the gpu_unit_tests are failing with this change?

FAILED MaxText/tests/decode_tests.py::DecodeTests::test_gpu_base - TypeError: ResultTokens.__init__() got an unexpected keyword argument 'log_prob'
FAILED MaxText/tests/decode_tests.py::DecodeTests::test_gpu_int8 - TypeError: ResultTokens.__init__() got an unexpected keyword argument 'log_prob'
FAILED MaxText/tests/decode_tests.py::DecodeTests::test_gpu_pdb_lt_1 - TypeError: ResultTokens.__init__() got an unexpected keyword argument 'log_prob'
FAILED MaxText/tests/maxengine_test.py::MaxEngineTest::test_basic_prefill - TypeError: ResultTokens.__init__() got an unexpected keyword argument 'log_prob'

[wenxindongwork]

Do you know why the gpu_unit_tests are failing with this change?

FAILED MaxText/tests/decode_tests.py::DecodeTests::test_gpu_base - TypeError: ResultTokens.__init__() got an unexpected keyword argument 'log_prob'
FAILED MaxText/tests/decode_tests.py::DecodeTests::test_gpu_int8 - TypeError: ResultTokens.__init__() got an unexpected keyword argument 'log_prob'
FAILED MaxText/tests/decode_tests.py::DecodeTests::test_gpu_pdb_lt_1 - TypeError: ResultTokens.__init__() got an unexpected keyword argument 'log_prob'
FAILED MaxText/tests/maxengine_test.py::MaxEngineTest::test_basic_prefill - TypeError: ResultTokens.__init__() got an unexpected keyword argument 'log_prob'

The Jetstream PR has to go through first, since it updates the ResultTokens API.

[vipannalla]

This PR includes changes from many other PRs which are already landed and its hard to read the specific code chagnes. Can you rebase your branch and update this PR?

[vipannalla]

Thanks, looks good