[New Feature]: Implement bal tool command

[gayaldassanayake]

Description

Implement a new CLI command bal tool that can be used to pull and manage ballerina related tools.

Resources

Design documents

1. https://docs.google.com/document/d/1fsDlGDRV1Ly2NjDWF4SQodFJJDGIc9ja7toCFQ31OQY/edit?usp=sharing
2. https://docs.google.com/document/d/1XsZK3UlqXv4TLEgRKVc0UBRRilLTmLZ1heDTmIRuylE/edit?usp=sharing

Implementation documents

1. https://docs.google.com/document/d/1lNytg9gEIAJOQI7aH6jSWBAvshpmElzXerj3RJuLCEw/edit?usp=sharing

Describe your problem(s)

The list of Ballerina commands that are categorized as "Other" continue to grow. Also most of these tools are used only in specific use cases and does not make sense to pack them with the distribution.

Describe your solution(s)

Use bal tool to on demand pull and use these tools. The tools should be allowed to be published as first class packages in the ballerina central.

Tasks for milestone 1 (update 6)

• Introduce tool packages - assignee @ShammiL
• Implement bal tool command - assignee @gayaldassanayake
• Redesign command implementation strategy - assignee @gayaldassanayake

Please refer to bal tool implementation task breakdown for the entire task list.

Related area

-> Other Area

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[gayaldassanayake]

The following has been implemented so far.

1. bal-tool.toml and its related classes
2. bal tool list, bal tool uninstall implementations
3. bal tool pull - partially. Need to complete once the Central APIs are available.
4. Negative test cases for list, uninstall, pull
5. BalTool.toml packing into the bala of a tool package.

[gayaldassanayake]

The following are the main tasks yet to be completed in bal tool implementation as of now.

High priority

• Implement the central API call methods for tool pulling - ongoing
• Mocking and manually testing bal tool pull - ongoing
• Implement bal tool search and test with mocking.
• Look how bal help flag can be dynamically extended when new tools are added.
• Pick the latest version when there are multiple versions of the same tool.

Mid priority

• Complete ToolCommand lang repo tests
• Complete bal tool help file
• Add test cases for BalTool.toml
• Add test cases for bal-tools.toml
• Add distribution repo integration tests for ToolCommand
• Create test scenario document

[gayaldassanayake]

With #40265 and #40457, the following are completed.

1. bal tool pull [:version] - Pull a tool from the central. If the version is not specified, the latest will be pulled.
2. bal tool list - List the latest versions of locally available tools
3. bal tool remove[:version] - Remove a tool from the local - If the version is not specified, all versions will be removed.
4. Introducing a BalTool.toml and packing and pushing a tool to the central

Next steps

• bal tool search - The implementation is completed. Waiting on the central endpoints to test and complete.
• bal tool use, bal tool update.
• Implement the proposed design for seamlessly switching between distribution versions and using tools.
• Remove class path loading from the shell script and implementing it in java.
• Introduce documentation
• Dynamically extend bal help with the available tools - Implemented. Need slight changes to the implementation

[ShammiL]

With #40475 the following were added.

1. Add a new template tool to be used with bal new. E.g.: bal new newTool -t tool
   • A BalTool.toml file specifying the tool id and an empty tool directory will be created inside the new package.
2. Allow using tools pushed to Central as templates when creating a new package. E.g.: bal new newTool -t foo/templateTool:1.0.0
   • Files packed in the tool will be copied to the new package including the BalTool.toml.
   • Dependencies packed with the tool will also be copied to tool/libs directory inside the new package.

[gayaldassanayake]

Currently working on following implementation changes/ improvements planned for the next U6 patch version.

1. Move the classloading to BLauncher.Main. Add the bre/lib, tool jar and dependencies to the classpath
2. Spin up a new process builder for tool commands only
3. Check if we can add debugging support
4. Providing a directory containing dependencies for BalTool [[dependency]] instead of individual jars - requirement by HealthCare.

[gayaldassanayake]

Update on current progress (U8)

1. The design has been revamped to have tool versions globally available. If a tool version is incompatible with a given distribution, there will be an error when trying to use pull, use commands or when executing the tool itself.
2. The bal help is automatically extended to show the help texts of the tools under Tool Commands section
3. https://ballerina.io/learn/cli-documentation/cli-commands/ is updated to include tool subcommands and its functionality
4. A separate custom class loader has been introduced to load tool commands giving priority to the dependencies of the tools over the system class loader resources.
5. openapi, grpc, graphql, persistandasyncapiare to be published as central packages and they will be pulled automatically when user callsbal openapi` etc, without user explicitly pulling them.

[Thevakumar-Luheerathan]

As a part of this effort, we are planning to add support for local repository. User may do the following tasks with this support,

1. Publish a tool to the local repository.
   bal push --repository=local

2. Pull a tool from a local repository.
   bal tool pull <tool-id>:<version> --repository=local

3. Activate a local repository tool from pulled tools.
   bal tool use <tool-id>:<version> --repository=local

4. Remove a local repository tool from pulled tools.
   bal tool use <tool-id>:<version> or bal tool remove <tool-id>:<version> --repository=local

Related issue : #41729

[gayaldassanayake]

Closing the issue since most of the original design has been implemented and successfully used by users. We can keep track of the improvements with separate dedicated issues.