[tools] package bookkeeper tools into a separated distribution

[sijie]

Descriptions of the changes in this PR:

Motivation

Server distribution contains a lot of stuffs that doesn’t need to run bkctl and bookkeeper shell, such as stats providers, http servers and many other dependencies.

At many environments, you want to deliver a set of tools to your customers to interact with or operate bookkeeper clusters. They don’t care about server side stuffs and ask them to use a server package to run a tool makes things confused.

changes

Create a new ‘bkctl’ package that only contains bookkeeper shell and new cli.

---

In order to uphold a high standard for quality for code contributions, Apache BookKeeper runs various precommit
checks for pull requests. A pull request can only be merged when it passes precommit checks. However running all
the precommit checks can take a long time, some trivial changes don't need to run all the precommit checks. You
can check following list to skip the tests that don't need to run for your pull request. Leave them unchecked if
you are not sure, committers will help you:

• [skip bookkeeper-server bookie tests]: skip testing org.apache.bookkeeper.bookie in bookkeeper-server module.
• [skip bookkeeper-server client tests]: skip testing org.apache.bookkeeper.client in bookkeeper-server module.
• [skip bookkeeper-server replication tests]: skip testing org.apache.bookkeeper.replication in bookkeeper-server module.
• [skip bookkeeper-server tls tests]: skip testing org.apache.bookkeeper.tls in bookkeeper-server module.
• [skip bookkeeper-server remaining tests]: skip testing all other tests in bookkeeper-server module.
• [skip integration tests]: skip docker based integration tests. if you make java code changes, you shouldn't skip integration tests.
• [skip build java8]: skip build on java8. ONLY skip this when ONLY changing files under documentation under site.
• [skip build java9]: skip build on java9. ONLY skip this when ONLY changing files under documentation under site.

---

---

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

If this PR is a BookKeeper Proposal (BP):

• Make sure the PR title is formatted like:
  <BP-#>: Description of bookkeeper proposal
e.g. BP-1: 64 bits ledger is support
• Attach the master issue link in the description of this PR.
• Attach the google doc link if the BP is written in Google Doc.

Otherwise:

• Make sure the PR title is formatted like:
  <Issue #>: Description of pull request
e.g. Issue 123: Description ...
• Make sure tests pass via mvn clean apache-rat:check install spotbugs:check.
• Replace <Issue #> in the title with the actual Issue number.

---

[eolivelli]

Good Idea. It will be useful

Does the script which checks licenses cover this new package?

[merlimat]

We could exclude this as it's only relevant to bookies

[sijie]

good point; excluded rocksdb

[sijie]

Does the script which checks licenses cover this new package?

travis checks this package.

[sijie]

run pr validation

[merlimat]

👍

[reddycharan]

Hey @sijie whats the purpose of this change? what are you moving and how would it help?

[sijie]

@reddycharan I am not moving anything. I am creating a new binary distribution that only includes tools. People can just download the tools package and use it to operate bookkeeper clusters without downloading the server package.

[reddycharan]

other files in this folder has 'bin' prefix,

bin-all.xml
bin-server.xml

may be you want to follow the same approach

[sijie]

bin-all and bin-server are two server distributions for "server binary" distributions. so that's why it was prefixed with bin (in the context, there was only one server binary distribution, the file was called bin).

calling bkctl is to differentiate the package from others.

[reddycharan]

@sijie i feel word 'tools' is kind of misleading. It kind of gives impression that it just contains tools to operate / administer bookkeeper like bookkeeper, bookkeeper-cli, bookkeeper http server stuff..reconsider the name for this package. And add more info. to this commit for what is the purpose for this package and how it differs from all/server package .

when you say "People can just download the tools package and use it to operate bookkeeper clusters without downloading the server package.", one would ask what is wrong in using server package?

[sijie]

@reddycharan updated the description. And if you have reviewed the PR, the package is not called ‘tools’, it is ‘bkctl’.

[reddycharan]

@sijie thanks for explanation. Yes, I was little confused about purpose of this package.

When you say "At many environments, you want to deliver a set of tools to your customers to interact with or operate bookkeeper clusters."

do you mean to say this would be for customers who wants to download this package to their control nodes (not bookie nodes) and run bookkeeper shell script? so they would be able to run only commands in 'cluster'/'ledger' group but not in 'bookie' (commands which access local filesystem of bookie) group?

[sijie]

so they would be able to run only commands in 'cluster'/'ledger' group but not in 'bookie' (commands which access local filesystem of bookie) group?

yes. ideally bookie commands should be removed or hidden from this tool, or those commands should be changed to access remote either via ssh tunneling or http rest endpoint to do the actual work. however that will be done in separate tasks.

the idea to have bkctl package is to start moving towards that direction: people can download a bkctl package/binary, set a service uri (e.g. zk://zkservers/ledgers) and start operating the cluster.

[reddycharan]

ok. lgtm.