[Improvement]: AMS server-expose-host should be adapted to domain name scenarios.

[baiyangtx]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

Currently, the value of AMS server-expose-host is an IP address, which must be readable from the network card.

This way of exposing the AMS access address only works when all components are in the same interconnected network environment. It cannot be used in other network structures, such as accessing through a domain name or with a proxy. This situation is quite common, such as in a K8S cluster. Therefore, it is necessary to redesign the way AMS exposes its access address.

The expose-host may be used in the following places:

1. When submitting to the Optimizer, it informs the Optimizer how to register itself with AMS.
2. In an HA scenario, it registers its access address with Zookeeper, including the Table Thrift address and the Optimizer Thrift address.
3. In InternalCatalog, it generates the Iceberg RestCatalog URI for client-side access.

We may need a new way to separately expose the HTTP access endpoint, Table Thrift endpoint, and Optimizer Thrift endpoint.

How should we improve?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[baiyangtx]

Regarding enabling AMS to correctly expose its access endpoints when deployed in a K8S environment, the following changes will be made to the config.

Currently:

ams:
  server-expose-host: "127.0.0.1"     # required, must be an ip address or ip address prefix.
  
  # the exposed port must be same with bind-port
  thrift-server:
    table-service:
      bind-port: 1260
    optimizing-service:
      bind-port: 1261
  http-server:
    bind-port: 1630

New plan:

ams:
  server-expose-address: "127.0.0.1"     # required, could be an ip address, ip address prefix, hostname, or domain.
  
  # the exposed port must be same with bind-port
  thrift-server:
    table-service:
      bind-port: 1260
      expose-endpoint: table.amoro.local:1260    # optional, could be an endpoint and will overwrite ams.server-expose-address
    optimizing-service:
      bind-port: 1261
      expose-endpoint: optimizing.amoro.local:1260    # optional, could be an endpoint and will overwrite ams.server-expose-address
  http-server:
    bind-port: 1630
    expose-endpoint: rest.amoro.local:1260    # optional, could be an endpoint and will overwrite ams.server-expose-address

Changes:

1. Rename ams.server-expose-host to ams.server-expose-address
2. ams.server-expose-address supports ip address, ip address prefix and hostname or a specific domain
3. Add optional ams.thrift-server.table-service.expose-endpoint, it will overwrite ams.server-expose-address if be configured.
4. Add optional ams.thrift-server.optimizing-service.expose-endpoint, it will overwrite ams.server-expose-address if be configured.
5. Add optional ams.http-server.expose-endpoint, it will overwrite ams.server-expose-address if be configured.
6. In HA mode, the endpoints of different services will be injected into Zookeeper separately. and in the properties connected to AMS, the type of service to be accessed needs to be specified.

[zhoujinsong]

Hi，It seems that opening separate endpoint configurations for each service can indeed solve the problem in the K8S environment.

However, in more complex network environments, different optimizers may need to connect to AMS using different URIs, so perhaps we can think about the problem from the perspective of the optimizer. For example, we can open configurations for different optimizer groups and set the address they use when connecting to AMS. I think this solution may be more flexible.

[baiyangtx]

Hi，It seems that opening separate endpoint configurations for each service can indeed solve the problem in the K8S environment.

However, in more complex network environments, different optimizers may need to connect to AMS using different URIs, so perhaps we can think about the problem from the perspective of the optimizer. For example, we can open configurations for different optimizer groups and set the address they use when connecting to AMS. I think this solution may be more flexible.

You're right, it may be more appropriate to support setting the access address from the perspective of the Optimizer Group.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'