Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiple redis destination #1275

Closed
simpleuser99 opened this issue Dec 8, 2016 · 5 comments
Closed

multiple redis destination #1275

simpleuser99 opened this issue Dec 8, 2016 · 5 comments

Comments

@simpleuser99
Copy link

@simpleuser99 simpleuser99 commented Dec 8, 2016

For one redis destination syslog-ng can only about 4000 events per second and this is very little for me. I need about 40k events syslog-ng to redis.
When I use syslog-ng 3.7 I use multiple destination

destination d_redis {
    redis(
        host("localhost")
        port(6379)
        command("rpush" "pp_rtest" "${DATE} ${HOST} ${MSGHDR}${MSG}")
        throttle(0)
    );
};

destination d_redis1 {
    redis(
        host("localhost")
        port(6379)
        command("rpush" "pp_rtest" "${DATE} ${HOST} ${MSGHDR}${MSG}")
        throttle(0)
    );
};

destination d_redis2 {
    redis(
        host("localhost")
        port(6379)
        command("rpush" "pp_rtest" "${DATE} ${HOST} ${MSGHDR}${MSG}")
        throttle(0)
    );
};

destination d_redis3 {
    redis(
        host("localhost")
        port(6379)
        command("rpush" "pp_rtest" "${DATE} ${HOST} ${MSGHDR}${MSG}")
        throttle(0)
    );
};

And multiple filter

filter f_redis     { "$(% ${PID} 4)"  eq "0"  };
filter f_redis1     { "$(% ${PID} 4)"  eq "1"  };
filter f_redis2     { "$(% ${PID} 4)"  eq "2"  };
filter f_redis3     { "$(% ${PID} 4)"  eq "3"  };
log { source(s_ext); filter(f_redis); destination(d_redis); };
log { source(s_ext); filter(f_redis1); destination(d_redis1); };
log { source(s_ext); filter(f_redis2); destination(d_redis2); };
log { source(s_ext); filter(f_redis3); destination(d_redis3); };

When I update syslog-ng to 3.8.1 version I have the next error:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.; persist_name='redis(localhost,6379)', location='/etc/syslog-ng/conf.d/pg_to_redis.conf:31:5'

How I can forwarder from syslog-ng to redis in one flow more than 4k events?
Or how I can use multiple redis destination like this I use it in syslog-ng ver 3.7 ?

@bazsi
Copy link
Collaborator

@bazsi bazsi commented Dec 8, 2016

@simpleuser99
Copy link
Author

@simpleuser99 simpleuser99 commented Dec 8, 2016

Just supply a unique persist-name option to redis, as recommended, and you
should be fine.

How? I have one redis in one ip adress and one port.

Also, if you are willing to do a perf record/report on Syslog-ng, we could
learn the reason it does only 4k/sec
How I can do it?

@simpleuser99
Copy link
Author

@simpleuser99 simpleuser99 commented Dec 8, 2016

You talked about persistent(yes) ? Right?

destination d_redis {
    redis(
        host("localhost")
        port(6379)
        command("rpush" "pp_rtest" "${DATE} ${HOST} ${MSGHDR}${MSG}")
        throttle(0)
        persistent(yes)
    );
};

destination d_redis1 {
    redis(
        host("localhost")
        port(6379)
        command("rpush" "pp_rtest" "${DATE} ${HOST} ${MSGHDR}${MSG}")
        throttle(0)
        persistent(yes)
    );
};

destination d_redis2 {
    redis(
        host("localhost")
        port(6379)
        command("rpush" "pp_rtest" "${DATE} ${HOST} ${MSGHDR}${MSG}")
        throttle(0)
        persistent(yes)
    );
};

destination d_redis3 {
    redis(
        host("localhost")
        port(6379)
        command("rpush" "pp_rtest" "${DATE} ${HOST} ${MSGHDR}${MSG}")
        throttle(0)
        persistent(yes)
    );
};
@juhaszviktor
Copy link
Collaborator

@juhaszviktor juhaszviktor commented Dec 8, 2016

@simpleuser99
Copy link
Author

@simpleuser99 simpleuser99 commented Dec 12, 2016

@juhaszviktor many thanks. This work or me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.