Connect Algorand node to the ELK stack [$850]

[michielmulders]

Overview

• Short Description: Create a tutorial that demonstrates how Algorand node runners can connect and use ELK stack to monitor their nodes.
• Bounty: $850 in USDCa or equivalent in ALGO tokens for non-US residents. (The amount of ALGO will be based on the 30-day average price of ALGO on the payment date) Recipient has to opt-in to receive USDCa asset on Algorand blockchain.
• Estimated Time Commitment: 2-3 Days
• Category: Development & Content
• Experience Level: Advanced

Description

What is this task?

Create a practical tutorial with working configuration files that demonstrates how to connect and use ELK stack with an Algorand node for monitoring purposes.

This includes:

• Orchestrating Elasticsearch, Logstash and Kibana instances.
• Configuring Logstash to ingest logs from the node
• Building a Kibana dashboard with interesting metrics, such as number of blocks proposed, number of Step 1 (soft) votes, and number of Step 2 (certify) votes (for each Algorand address on behalf of which the node participate), log severity distribution, etc.

What are the requirements for the bounty taker?

The bounty taker should have the knowledge of how to run a node, read and understand its logs. In addition, experience in setting up and operating Elasticsearch, Logstash and Kibana is required.

What are the deliverables?

1. GitHub project

A GitHub project with all necessary configuration files and the exported Kibana dashboard.

Judging Criteria and Metrics

• All files must be compatible with the versions of ELK used in the tutorial.
• Configuration files must be clean, well-documented, and secure.
• README file with a comprehensive explanation and project structure.

Submission Procedure

• Submit the project link on the Gitcoin bounty thread in GitHub

2. Tutorial

Submit a tutorial describing all steps on the Algorand Developer Portal.

Judging Criteria and Metrics

• Blog submissions will be evaluated by the Algorand Community team
• Include code snippets using markdown code blocks
• Add images where relevant and submit the source files for these images
• Check your blog post with Grammarly

Submission Procedure

• Submit your blog post following these steps:

• Create an account in the Developer Portal
• Complete your profile
• Select Blog from the left sidebar in Admin Dashboard
• Click New Blog Post
• Fill in the title and content. Upload a featured image.
• When you’re done writing, click Save as Draft to preview or click Submit to publish
• Check the appropriate boxes in the authorization modal and click Submit
• Send a message on the Gitcoin bounty that you've published a blog post together with your username so the Algorand team can review it

Other Requirements

For questions, reach out to Algorand on Discord.

• Taking this bounty task means you have agreed to our Bounty Agreement terms and policies.
• All project codes should be published on GitHub, with MIT license or Apache V2 license.
• All contributions/PRs to public (Algorand) repositories should be open source and should follow the contributions rules of the relevant repository.
• All non-codes documents should be using Markdown format.
• In the project description or repository readme, there should be a proper security notice. E.g., it should tell the visitor the project is not audited and should not be used in a production environment.
• You are not eligible to receive additional rewards through other Algorand programs like the developer ambassadors program.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 849.9953 USDCa (858.5 USD @ $1.01/USDCa) attached to it as part of the algorandfoundation fund.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin's Discord
• $1,944,299.81 more funded OSS Work available on the Gitcoin Issue Explorer

[michielmulders]

@ballagoes Can you introduce yourself and your experience with Algorand/ELK stack to see if this is a good match? If yes, you can re-apply for the bounty. Thank you!

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Workers have applied to start work.

These users each claimed they can complete the work by 265 years, 5 months from now.
Please review their action plans below:

1) dozham has applied to start work (Funders only: approve worker | reject worker).

Hello there,
Hope all is well for you.
In fact, I am really thrilled by this issue and the required task!
I have 3+ years experience working with the ELK stack handling TBs of data every day.
Specifically, I have developed log-based monitoring/alerting systems that ingest data from several microservices. Maintenance and development of such systems is part of what I do on a daily basis.
The steps seem pretty straightforward as the instructions on running Algorand nodes is well-documented by the foundation before. At first, I will complete the required steps on my own, and then I will prepare the detailed steps and tutorials as the next step:

1. Brief explanation of the required software (ELK, Algorand nodes) and the objective of the tutorial.
2. Software/hardware requirements of the tutorial and where to find the required information on their installation/running.
3. How to configure and connect ELK to ingest data from Algorand nodes.
4. How to build Kibana dashboards to provide insightful data on the status of the nodes.
5. Recommnedations on different steps on what concerns should be noted to maintain performance/security when using in production.

My only minor concern is that I am not highly experienced with Algorand. But, I do have some domain-knowledge on cryptocurrencies, concensus algorithms and corresponding protocols. So, I wanted to ask if you think I should take some time reading Algorand documents. If so, can you provide any list of pages/content that can help me start the task quickly?
And, on the metrics that you have suggested; they seem pretty straightforward to implement (using simple Elasticsearch/KQL queries), but if you think other interesting metrics can be defined to highlight the value of the task, I would be more than happy to hear your suggestions (due to your Algorand expertise).

Well, that's it. Hopefully I can be of assistance.
Looking forward to hearing from you.
2) stylishsquid has applied to start work (Funders only: approve worker | reject worker).

1. Build out an ELK stack, using the easiest configuration.
2. Setup a Algo node with logging on.
3. Ingest algo logs: file beat -> logstash -> es
4. Configure logstash filter to parse out and enrich log data. This will make it easier to build kibana dashbaord.
5. Configure kibana, and build dashbaord. Dashboard should be easy if logstash enriching is done properly.
6. Documentation/Instructions/Blog
   6a. For the setup of Algo node and ELK stack would be references to other blog posts out there, or maybe create a condensed version of those instructions here.
   6b. Break down what is in an Algo log. What's the format. This will help in building logstash filter.
   6c. How to write up a logstash filter to enrich Alog logs.
   6d. How to building of kibana dashboard. This will bring together step 6b and 6c, or understanding the logs and enriching them.
   6e. Showing other examples of how to query Algo logs.

Learn more on the Gitcoin Issue Details page.

[michielmulders]

@dozham Thank you for your application. I like your strong experience with the ELK stack, however, I'm a bit concerned about the low Algorand knowledge. We can definitely guide you on this project, especially on the metrics we would like to see. Setting up a node is relatively straightforward.

If you're ok with this, I'll leave the bounty open for other people to apply who might have a deeper understanding of the Algorand ecosystem/nodes. It's a very specific bounty, so let's see. We can definitely guide you but we prefer to have as much as possible independent bounty hunters. Thanks!

[dozham]

@michielmulders Thanks for your prompt response.
I am totally OK with your decision.
In the mean time, I will get myself more familiar with Algorand ecosystem.

Best regards.

[michielmulders]

@dozham Thank you for understanding. We'll be launching more bounties (from beginner to advanced) these weeks, so there's definitely something you can pick up. Make sure to follow my Gitcoin account to get notified when a new bounty is published.

[michielmulders]

@StylishSquid Thank you for applying. Can you tell me a bit more about your experience with Algorand? I like the steps in your overview, however, I'm missing some information about the Kibana dashboard? We would like to see specific metrics about votes, blocks, transactions, consensus, etc. Thanks for your time!

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

The funding of 849.9953 USDCa (850.0 USD @ $1.0/USDCa) attached to this issue has been cancelled by the bounty submitter

• Questions? Checkout Gitcoin Help or the Gitcoin's Discord
• $1,944,503.21 more funded OSS Work available on the Gitcoin Issue Explorer

[michielmulders]

@dozham @StylishSquid We had to move the bounty due to an issue on our side. I'll keep you posted. Feel free to reapply already so we can approve you later. Here's the updated bounty. Sorry for the inconvenience!