Skip to content
This repository has been archived by the owner on Jan 8, 2019. It is now read-only.

serve conf pages on the cluster's virtual ip address #1244

Closed
wants to merge 3 commits into from

Conversation

leochen4891
Copy link
Contributor

This is to make conf pages HA available on the cluster's virtual ip.
Each each conf page is identified by the port number.
e.g.
HDFS: http://10.0.100.5:50071/conf
HBase: http://10.0.100.5:16011/conf

More pages will be added in the future (e.g. Yarn)

Ready for review.

@leochen4891
Copy link
Contributor Author

Will add port 60301 for hbase workers and 1006 for hdfs workers.

@cbaenziger
Copy link
Member

Please use the standard ports and please do not make up a new convention. We already have enough confusion with the accident of 11010. Nothing should be listening on the VIP for 16010, 60300, nor 50070.

@leochen4891
Copy link
Contributor Author

yes, it make sense. I have observed some issue that HBase is binding to 0.0.0.0 which should be fixed first.

option httpchk GET /jmx
http-check expect string tag.isActiveMaster"\ :\ "true
<% @hbase_servers.each do |server| -%>
<%= "server #{server['hostname']} #{server['bcpc']['management']['ip']}:16010 check" %>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get the port number from an attribute?

option httpchk GET /jmx
http-check expect string tag.HAState"\ :\ "active
<% @hdfs_servers.each do |server| -%>
<%= "server #{server['hostname']} #{server['bcpc']['floating']['ip']}:50070 check" %>
Copy link
Contributor

@amithkanand amithkanand Aug 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NN port number from an attribute? Also, all other ports in haproxy.

@@ -74,3 +76,21 @@ backend backend_oozie
<%= "server #{float_host(server[:fqdn])} #{server['bcpc']['floating']['ip']}:11000 check" %>
<% end -%>
<% end -%>

# hdfs conf page
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not call this the conf page but instead hdfs master or some such

@leochen4891
Copy link
Contributor Author

may need to redirect.

@bijugs
Copy link
Contributor

bijugs commented Aug 3, 2018

Updates to haproxy should be made from bcpc-hadoop cookbook since these are for the services deployed by bcpc-hadoop. The changes as it is creates an in direct dependency for bcpc cookbook on bcpc-hadoop. For e.g. if NN port gets changed in bcpc-hadoop, then bcpc cookbook needs to be updated which is not desirable.

@leochen4891
Copy link
Contributor Author

working on remove the circular dependency between bcpc:haproxy and bcpc-hadoop ...

@leochen4891
Copy link
Contributor Author

Services should not bind to vip (#1258)
Afterwards, haproxy can use the standard port number associated to a service, e.g. hbase = 16010

Also, as Biju has pointed out, circular dependency should be avoided. I'm going to add attributes in haproxy cookbook which are going to be overridden in bcpc-hadoop cookbook.

@pu239ppy
Copy link
Contributor

@leochen4891 For simplicity's sake you could have had an haproxy configuration that listens on some designated port XXXX and elected to use a conventions such as conf-service:XXXX/hbase-master,
conf-service:XXXX/hdfs-datanode, conf-service:XXXX/yarn-nodemanager

@leochen4891
Copy link
Contributor Author

Thanks for the suggestions. I have put the change to a new PR. #1269

Closing this one.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants