[serve] Remove BackendConfig broadcasting

[edoakes]

Why are these changes needed?

Depends on: #19145

Instead of broadcasting max_concurrent_queries at the deployment level, tacks it on as additional metadata for each individual replica.

Related issue number

Closes #19147

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[edoakes]

@simon-mo review #19145 first

[simon-mo]

Yup. Totally agree this is the right track. For the Java backend prototype, I ended up adding a similar structure as ReplicaInfo

[edoakes]

cc @jiaodong discovered this while working on this PR -- the user_config hashes are different across processes, so the recovery logic is likely not working for these...

[jiaodong]

wow this was quite subtle but great catch. Let's follow up more in slack as it seems we need more thorough testing and harding around FT stuff.

[simon-mo]

let's categorize this as a tech debt and add an issue to the tech debt milestone? setting PYTHONHASHSEED is generally bad as this might also set it for whatever user code depends on

Hash randomization is intended to provide protection against a denial-of-service caused by carefully-chosen inputs that exploit the worst case performance of a dict construction, O(n2) complexity. See http://www.ocert.org/advisories/ocert-2011-003.html for details.

[edoakes]

@simon-mo take a look at the updated pickle.dumps version?

[edoakes]

@simon-mo this is ready for review now

[simon-mo]

LGTM. Two minor comments about replica.py code changes.