[FEATURE] OPT-175B service authentication and new priority queue

[JubilantJerry]

See the issue page #700. I made some design choices based on my comment there, see if the design choices make sense.

[JubilantJerry]

Example informal test that I've done:

I set the weight for API key "key1" to be default (10) and "key2" to be 3 times higher (30).

I've run the following on the command line

for i in $(seq 0 20); do (sleep 0.$[100 + ($RANDOM % 900)];  curl -d '{"model": "default", "prompt": ["This year,", "This month,", "This day,", "Today, we"], "max_tokens": 16, "api_key": "key2"}' -H 'Content-Type:application/json' localhost:20001/completions -s > /dev/null && echo "AAAAAAA$i" ) & done
for i in $(seq 0 20); do (sleep 0.$[100 + ($RANDOM % 900)];  curl -d '{"model": "default", "prompt": ["This year,", "This month,", "This day,", "Today, we"], "max_tokens": 17, "api_key": "key1"}' -H 'Content-Type:application/json' localhost:20001/completions -s > /dev/null && echo "BBBBBBB$i" ) & done
for i in $(seq 0 20); do (sleep 0.$[100 + ($RANDOM % 900)];  curl -d '{"model": "default", "prompt": ["This year,", "This month,", "This day,", "Today, we"], "api_key": "key1", "top_k": 10}' -H 'Content-Type:application/json' localhost:20001/logprobs -s > /dev/null && echo "CCCCCCC$i" ) & done
for i in $(seq 0 20); do (sleep 0.$[100 + ($RANDOM % 900)];  curl -d '{"model": "default", "prompt": ["This year,", "This month,", "This day,", "Today, we"], "max_tokens": 18}' -H 'Content-Type:application/json' localhost:20001/completions -s > /dev/null && echo "DDDDDDD$i" ) & done
for i in $(seq 0 20); do (sleep 0.$[100 + ($RANDOM % 900)]; curl -d '{"model": "default", "prompt": ["This year,", "This month,", "This day,", "Today, we"], "top_k": 10}' -H 'Content-Type:application/json' localhost:20001/logprobs -s > /dev/null && echo "EEEEEEE$i" ) & done

At the same time, I start making requests with several tabs on the web interface.

The result is that:

1. The web interface requests are handled with low latency,
2. The AAAAAAA requests are served about 3 times more often than the BBBBBBB requests and CCCCCCC requests,
3. The DDDDDDD and EEEEEEE requests are served about 10 times less often than the BBBBBBB requests,

[merrymercy]

Thanks for your contribution. I replied to your comments. I will review this PR later this week.

[merrymercy]

Looks like the code is well-documented and well-tested. The design is what we want in #700 and it carefully considers a lot of potential issues.
Thanks for your contribution! I will do more tests and try to deploy it on our website. I will merge this first as the code holds very high quality.

[JubilantJerry]

That's good to hear, I haven't used automated test suites unlike the rest of Alpa but hopefully the informal tests are good enough coverage. I realize that one of the comments is wrong, but that can be fixed elsewhere (I'll include it in #870 in case that feature gets added)

[JubilantJerry]

Found an error in the comment, there's a separate queue for each authentication choice, not for the pair of authentication choice and endpoint