Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logstash persistent queue improvement #2744

Merged
merged 2 commits into from Feb 15, 2021

Conversation

PMazarovich
Copy link
Contributor

Motivation and context

In current implementation of ELK there is no persistent queue in logstash. This can lead to the loss of important analytics data. With this improvement any data, that was sent by cvat will eventually be sent into logstash if logstash instance was unavaliable for some time. Or to elasticsearch instance if it was unavaliable. So, all of features of persistent queue is now ready for working with cvat.
As a side note, there is a metadata with year and month added. This is done for better flexibility in index management in elasticsearch.

How has this been tested?

It was tested manually

Checklist

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below)
# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

@coveralls
Copy link

coveralls commented Jan 29, 2021

Coverage Status

Coverage remained the same at 69.756% when pulling 86b1a4e on PMazarovich:logstash_pq into e382313 on openvinotoolkit:develop.

@azhavoro
Copy link
Contributor

azhavoro commented Feb 7, 2021

@PMazarovich Could you please merge current develop changes (or rebase, as you prefer) to fix CI build?

@PMazarovich
Copy link
Contributor Author

Rebase done

'host': os.getenv('DJANGO_LOG_SERVER_HOST', 'localhost'),
'port': os.getenv('DJANGO_LOG_SERVER_PORT', 5000),
'port': os.getenv('DJANGO_LOG_SERVER_PORT', 8080),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason to change the port from 5000 to 8080?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For http protocol, port 8080 is standard one, and logstash now works via http because of persistent queue

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@azhavoro , we use 8080 for cvat_proxy. Can it be a problem?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not open port, I don't see any problem here. We use the same port number for cvat_server https://github.com/openvinotoolkit/cvat/blob/develop/Dockerfile#L157

@nmanovic nmanovic merged commit 7720a8f into cvat-ai:develop Feb 15, 2021
@nmanovic
Copy link
Contributor

@PMazarovich , thanks for the contribution!

kenu pushed a commit to kenu/cvat that referenced this pull request Feb 23, 2021
Co-authored-by: Andrey Zhavoronkov <andrey.zhavoronkov@intel.com>
@chaoyifei
Copy link

HI, the log in the rq like INFO:cvat.server_33:Create Task #33 ,Uploadpath ... is not into logstash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants