Index channel logs in elasticsearch #89

Closed
wants to merge 3 commits into
from

Conversation

Projects
None yet
2 participants
@D-Ne0
Contributor

D-Ne0 commented Apr 30, 2014

A very basic working version of indexing channel logs in elasticsearch. Let me know what changes are required in this pull request.

sudo yum install elasticsearch
sudo service elasticsearch start
npm install elasticsearch

To test if elasticsearch is running, open: http://localhost:9200

To see all indexed documents

curl -XGET 'localhost:9200/channel_logs/_search?pretty=true'
@rtnpro

This comment has been minimized.

Show comment
Hide comment
@rtnpro

rtnpro May 3, 2014

Member

@D-Ne0 looks good 👍 . Also, please add some basic code to start pushing channel logs to ES.

Member

rtnpro commented May 3, 2014

@D-Ne0 looks good 👍 . Also, please add some basic code to start pushing channel logs to ES.

@D-Ne0

This comment has been minimized.

Show comment
Hide comment
@D-Ne0

D-Ne0 May 3, 2014

Contributor

@rtnpro The code in this pull request does the same. Do you want me to push old channel logs in mongo db too to ES? If yes, it's a one time task so should I write a script?

Contributor

D-Ne0 commented May 3, 2014

@rtnpro The code in this pull request does the same. Do you want me to push old channel logs in mongo db too to ES? If yes, it's a one time task so should I write a script?

@rtnpro

This comment has been minimized.

Show comment
Hide comment
@rtnpro

rtnpro May 18, 2014

Member

@D-Ne0 Please add some settings variable to decide whether to use ES or not.

Member

rtnpro commented May 18, 2014

@D-Ne0 Please add some settings variable to decide whether to use ES or not.

@D-Ne0

This comment has been minimized.

Show comment
Hide comment
@D-Ne0

D-Ne0 May 18, 2014

Contributor

Done

Contributor

D-Ne0 commented May 18, 2014

Done

waartaa/server/lib/02_irc.js
@@ -214,6 +214,17 @@ ChannelLogsManager = function () {
function _insert (log) {
ChannelLogs.insert(log, function (err, id) {});
OldChannelLogs.insert(log, function (err, id) {});
+ if(CONFIG.ENABLE_ELASTIC_SEARCH) {

This comment has been minimized.

@rtnpro

rtnpro May 26, 2014

Member

why no space after if?

@rtnpro

rtnpro May 26, 2014

Member

why no space after if?

waartaa/server/lib/02_irc.js
@@ -214,6 +214,17 @@ ChannelLogsManager = function () {
function _insert (log) {
ChannelLogs.insert(log, function (err, id) {});
OldChannelLogs.insert(log, function (err, id) {});
+ if(CONFIG.ENABLE_ELASTIC_SEARCH) {
+ var es_client = elasticsearch.Client({
+ host: 'localhost:9200'

This comment has been minimized.

@rtnpro

rtnpro May 26, 2014

Member

keep host configurable, in CONFIG

@rtnpro

rtnpro May 26, 2014

Member

keep host configurable, in CONFIG

waartaa/server/lib/02_irc.js
@@ -214,6 +214,17 @@ ChannelLogsManager = function () {
function _insert (log) {
ChannelLogs.insert(log, function (err, id) {});
OldChannelLogs.insert(log, function (err, id) {});
+ if(CONFIG.ENABLE_ELASTIC_SEARCH) {
+ var es_client = elasticsearch.Client({

This comment has been minimized.

@rtnpro

rtnpro May 26, 2014

Member

Let's use horntail notation, which is conventional in JS, e.g., esClient

@rtnpro

rtnpro May 26, 2014

Member

Let's use horntail notation, which is conventional in JS, e.g., esClient

@rtnpro

This comment has been minimized.

Show comment
Hide comment
@rtnpro

rtnpro May 26, 2014

Member

@D-Ne0

Rebase your code on top of latest master. Does https://gist.github.com/D-Ne0/10a479fd4b6c1c0f4c93 need to go in the ES server?

Member

rtnpro commented May 26, 2014

@D-Ne0

Rebase your code on top of latest master. Does https://gist.github.com/D-Ne0/10a479fd4b6c1c0f4c93 need to go in the ES server?

@D-Ne0

This comment has been minimized.

Show comment
Hide comment
@D-Ne0

D-Ne0 May 26, 2014

Contributor

Hey, I have rebased my code and tested the changes. It's working fine on my machine.

Not exactly sure what you meant by gist going into the ES server. As I have mentioned in comment we basically have to run this after ES is installed: https://gist.github.com/D-Ne0/10a479fd4b6c1c0f4c93#comment-1234856

Contributor

D-Ne0 commented May 26, 2014

Hey, I have rebased my code and tested the changes. It's working fine on my machine.

Not exactly sure what you meant by gist going into the ES server. As I have mentioned in comment we basically have to run this after ES is installed: https://gist.github.com/D-Ne0/10a479fd4b6c1c0f4c93#comment-1234856

@rtnpro

This comment has been minimized.

Show comment
Hide comment
@rtnpro

rtnpro Jun 1, 2014

Member

@D-Ne0 Rebase this as well.

Member

rtnpro commented Jun 1, 2014

@D-Ne0 Rebase this as well.

@rtnpro

This comment has been minimized.

Show comment
Hide comment
@rtnpro

rtnpro Jul 27, 2014

Member

Closing this as all changes merged to bookmarking branch.

Member

rtnpro commented Jul 27, 2014

Closing this as all changes merged to bookmarking branch.

@rtnpro rtnpro closed this Jul 27, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment