Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fromhost and fromhost-ip properties are not available to imfile #545

Open
ymattw opened this issue Oct 6, 2015 · 42 comments
Open

fromhost and fromhost-ip properties are not available to imfile #545

ymattw opened this issue Oct 6, 2015 · 42 comments

Comments

@ymattw
Copy link
Contributor

ymattw commented Oct 6, 2015

According to http://www.rsyslog.com/doc/master/configuration/properties.html, fromhost and fromhost-ip should be general properties, however, when I use them in a template for imfile module, they are expanded to empty. Is this a known issue? Thanks.

My template definition

template(name="format_json" type="list") {
    constant(value="{\"hostname\":\"")
    property(name="hostname")
    constant(value="\"")

    constant(value=",\"fromhost-ip\":\"")
    property(name="fromhost-ip")
    constant(value="\"")

    constant(value=",\"app-name\":\"")
    property(name="app-name" caseConversion="lower")
    constant(value="\"")

    constant(value=",\"filename\":\"")
    property(name="$!metadata!filename")
    constant(value="\"")

    constant(value=",\"syslogtag\":\"")
    property(name="syslogtag")
    constant(value="\"")

    constant(value=",\"message\":\"")
    property(name="msg" format="json")
    constant(value="\"")

    constant(value=",\"timestamp\":\"")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value="\"")

    constant(value="}")
}

Ruleset definition

module(load="omkafka")

ruleset(name="rule_omkafka") {
    action(type="omkafka"
           template="format_json"
           broker=["broker01.example.com:9092", "broker02.example.com:9092"]
           partitions.auto="on"
           topic="applog"

           action.resumeRetryCount="-1"
           queue.filename="queue"
           queue.size="100000"
           queue.saveonshutdown="on"
           queue.discardseverity="0"
          )
}

Input definition

module(load="imfile" mode="inotify")

input(type="imfile"
      ruleset="rule_omkafka"
      file="/export/logs/myapp.log"
      addmetadata="on"
      tag="myapp"
)
@rgerhards
Copy link
Member

Oh, if that's really a case, that's a bug. They should be populated with the local machine info.

@ymattw
Copy link
Contributor Author

ymattw commented Oct 11, 2015

Just reproduced with more simple configs: read from file and output to file (in json format). What could be wrong?

# more *conf
::::::::::::::
00-template.conf
::::::::::::::
template(name="format_json" type="list") {
    constant(value="{\"fromhost\":\"")
    property(name="fromhost")
    constant(value="\"")

    constant(value=",\"fromhost-ip\":\"")
    property(name="fromhost-ip")
    constant(value="\"")

    constant(value=",\"app-name\":\"")
    property(name="app-name" caseConversion="lower")
    constant(value="\"")

    constant(value=",\"filename\":\"")
    property(name="$!metadata!filename")
    constant(value="\"")

    constant(value=",\"syslogtag\":\"")
    property(name="syslogtag")
    constant(value="\"")

    constant(value=",\"message\":\"")
    property(name="msg" format="json")
    constant(value="\"")

    constant(value=",\"timestamp\":\"")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value="\"")

    constant(value="}")
}

::::::::::::::
02-ruleset-omfile.conf
::::::::::::::
module(load="builtin:omfile")

ruleset(name="rule_omfile") {
    action(type="omfile"
           dirCreateMode="0700"
           FileCreateMode="0644"
           template="format_json"
           File="/tmp/omfile.log"
    )
}

::::::::::::::
10-input-default.conf
::::::::::::::
module(load="imfile" mode="inotify")

input(type="imfile"
      ruleset="rule_omfile"
      addmetadata="on"
      tag="myapp"
      file="/tmp/input.log"
)

# echo TEST >> /tmp/input.log

# cat /tmp/omfile.log
{"fromhost":"","fromhost-ip":"","app-name":"myapp","filename":"/tmp/input.log","syslogtag":"myapp","message":"TEST","timestamp":"2015-10-11T12:45:20.456480+00:00"}

This was done from a docker container:

# cat /etc/hosts
172.17.0.54     rtest
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

# hostname
rtest

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
111: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:36 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.54/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:36/64 scope link
       valid_lft forever preferred_lft forever

I also did the same test on a virtualbox vm, saw the same.

@rgerhards
Copy link
Member

Imfile has a bug

@ymattw
Copy link
Contributor Author

ymattw commented Oct 11, 2015

If you tell more detail I could probably jump in to fix it :)

@rgerhards
Copy link
Member

I am on the road so hard to type much. Look at where the message is created (msgconstruct()?). I guess the properties are not set. Compare that to other plugins like imptcp.

@alorbach alorbach self-assigned this Oct 29, 2015
@alorbach
Copy link
Member

From what I can see MsgSetHOSTNAME is properly called, so the properties should be set.
I will run some tests on my dev environment to check this.

@alorbach
Copy link
Member

Could not reproduce the problem, works as expected with current master-candidate version:

{"hostname":"ubuntu","fromhost-ip":"127.0.0.1","app-name":"rsyslogd","filename":"","syslogtag":"rsyslogd:","message":" [origin software="rsyslogd" swVersion="8.14.0.master" x-pid="36711" x-info="http://www.rsyslog.com"] start","timestamp":"2015-10-30T04:33:11.860421-07:00"}

@ymattw
Copy link
Contributor Author

ymattw commented Dec 2, 2015

Interesting... I can still reproduce this with v8.14.0 on CentOS Linux release 7.1.1503 (Core), from a VM installed on MacOS (NOT inside docker container).

I use static linking with dependencies, with following configure parameters, anything wrong?

--prefix /export/servers/rsyslog \
--enable-static \
--enable-debug \
--enable-elasticsearch \
--enable-elasticsearch-tests \
--enable-liblogging-stdlog \
--enable-imfile \
--enable-imptcp \
--enable-omstdout \
--enable-omruleset \
--enable-omuxsock \
--enable-omkafka \
--disable-libgcrypt \
CFLAGS=-DPATH_PIDFILE='"/export/servers/rsyslog/var/run/rsyslog.pid"' \
PKG_CONFIG_PATH=/tmp/lib/pkgconfig \
LIBESTR_LIBS=/tmp/lib/libestr.a \
JSON_C_LIBS=/tmp/lib/libjson-c.a \
ZLIB_LIBS=/tmp/lib/libz.a \
LIBUUID_LIBS=/tmp/lib/libuuid.a \
CURL_LIBS=/tmp/lib/libcurl.a \
LIBLOGGING_STDLOG_LIBS=/tmp/lib/liblogging-stdlog.a \
LIBRDKAFKA_CFLAGS=-I/tmp/include \
LIBRDKAFKA_LIBS=/tmp/lib/librdkafka.a

Dependencies are:

cf46112b5151e2f1a3fd38439bdade23  curl-7.44.0.tar.gz
39705ae46b1c0c64f1d32d26653c8e7e  json-c-0.12-20140410.tar.gz
f4c9165a23587e77f7efe65d676d5e8e  libestr-0.1.10.tar.gz
44b8ce2daa1bfb84c9feaf42f9925fd7  liblogging-1.0.5.tar.gz
1b77543f9be82d3f700c0ef98f494990  librdkafka-0.8.6.tar.gz
d44d866d06286c08ba0846aba1086d68  libuuid-1.0.3.tar.gz
44d667c142d7cda120332623eab69f40  zlib-1.2.8.tar.gz

@ymattw
Copy link
Contributor Author

ymattw commented Dec 2, 2015

Also reproduced on a VM of version CentOS release 6.4 (Final), which has only 1 network interface (which I thought could be related initially).

@alorbach
Copy link
Member

Hrm I think this can only happen if the gethostname() api returns a failure. Unfortunetally rsyslog does not print an error into the debug log if this happens.

I am going to try to reproduce this problem on Centos6 tomorrow. I have a working vmware somewhere ;)

@alorbach alorbach added this to the v8.15 release milestone Dec 10, 2015
@ymattw
Copy link
Contributor Author

ymattw commented Jan 6, 2016

Weird .. I reproduced this on ubuntu 14.04.3 as well, from inside docker container, where gethostname() works totally fine.

With gdb I see the call trace is as below, the resolveDNS() call does nothing real because pMsg->msgFlags is always 0. I tried git grep that flag but got lost soon. This is low priority for me as I have an easy workaround (writing the values in template with a startup script), hope you can figure out what's going wrong.

(gdb) bt
#0  resolveDNS (pM=0x7ffff0005770) at msg.c:479
#1  getRcvFromIP (pM=0x7ffff0005770) at msg.c:523
#2  0x000000000041f51f in MsgGetProp (pMsg=0x7ffff0005770, pTpe=0x6d37a0, pProp=0x6d37c0, pPropLen=0x7ffff5bcfb0c, pbMustBeFreed=0x7ffff5bcfb1e, ttNow=<value optimized out>)
    at msg.c:3264
#3  0x00000000004460d1 in tplToString (pTpl=0x6d2d20, pMsg=0x7ffff0005770, iparam=0x7fffec002080, ttNow=0x7ffff5bcfbd0) at ../template.c:195
#4  0x0000000000441e1b in prepareDoActionParams (pAction=0x6d0170, pWti=0x6d6ed0, pMsg=0x7ffff0005770, ttNow=0x7ffff5bcfbd0) at ../action.c:937
#5  processMsgMain (pAction=0x6d0170, pWti=0x6d6ed0, pMsg=0x7ffff0005770, ttNow=0x7ffff5bcfbd0) at ../action.c:1303
#6  0x0000000000442481 in processBatchMain (pVoid=0x6d0170, pBatch=0x6d6f08, pWti=0x6d6ed0) at ../action.c:1347
#7  0x00000000004398b4 in ConsumerReg (pThis=0x6d0710, pWti=0x6d6ed0) at queue.c:1898
#8  0x0000000000433cf5 in wtiWorker (pThis=0x6d6ed0) at wti.c:334
#9  0x000000000043284e in wtpWorker (arg=0x6d6ed0) at wtp.c:389
#10 0x00007ffff7bc7a51 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffff750893d in clone () from /lib64/libc.so.6
(gdb) l
474             prop_t *localName;
475             DEFiRet;
476
477             MsgLock(pMsg);
478             CHKiRet(objUse(net, CORE_COMPONENT));
479             if(pMsg->msgFlags & NEEDS_DNSRESOL) {
480                     localRet = net.cvthname(pMsg->rcvFrom.pfrominet, &localName, NULL, &ip);
481                     if(localRet == RS_RET_OK) {
482                             /* we pass down the props, so no need for AddRef */
483                             MsgSetRcvFromWithoutAddRef(pMsg, localName);
(gdb) p pMsg->msgFlags
$18 = 0
(gdb) fin
Run till exit from #0  resolveDNS (pM=0x7ffff0005770) at msg.c:493
524                     if(pM->pRcvFromIP == NULL)
(gdb) p pM->pRcvFromIP
$20 = (prop_t *) 0x0
(gdb)

(gdb) fin
Run till exit from #0  getRcvFromIP (pM=0x7ffff0005770) at msg.c:524
0x000000000041f51f in MsgGetProp (pMsg=0x7ffff0005770, pTpe=0x6d37a0, pProp=0x6d37c0, pPropLen=0x7ffff5bcfb0c, pbMustBeFreed=0x7ffff5bcfb1e, ttNow=<value optimized out>)
    at msg.c:3264
3264                            pRes = getRcvFromIP(pMsg);
Value returned is $21 = (uchar *) 0x496a5d ""

PS: it's super easy to setup such a test env with docker, and here are the list I installed on top of en empty ubuntu:14.04 image to build rsyslog (from .bash_history):

apt-get install build-essential
apt-get install autoconf
apt-get install libtool
apt-get install zlib1g-dev
apt-get install pkg-config
apt-get install flex
apt-get install bison
apt-get install python-docutils

@elain
Copy link

elain commented Jan 13, 2016

Has this problem been solved? I've had the same problem.

@zachzhaozy
Copy link
Contributor

@elain Before the problem is fixed, it can be a temporary solution to use a placeholder in template file, and replace it with external IP by start script.

external_ip=$(echo $(/sbin/ip route get 1.1.1.1 | sed -n 's/.* src //p'))

Taught by @ymattw :-)

@rgerhards rgerhards modified the milestones: v8.33, v8.34 Feb 15, 2018
@rgerhards rgerhards modified the milestones: v8.34, v8.35 Apr 3, 2018
@rgerhards rgerhards modified the milestones: v8.35, v8.36 May 14, 2018
@rgerhards rgerhards modified the milestones: v8.36, v8.37 Jun 25, 2018
@rgerhards rgerhards modified the milestones: v8.37, v8.38 Aug 3, 2018
@rgerhards rgerhards modified the milestones: v8.38, v8.39 Sep 18, 2018
@rgerhards rgerhards modified the milestones: v8.39, v8.40 Oct 26, 2018
@rgerhards
Copy link
Member

does anybody still experience this issue? It looks a bit like it went away with recent imfile refactoring.

@rgerhards rgerhards removed this from the v8.40 milestone Oct 26, 2018
@deoren
Copy link
Contributor

deoren commented Nov 13, 2018

@rgerhards: does anybody still experience this issue? It looks a bit like it went away with recent imfile refactoring.

I'll try to make some time to test this and provide feedback.

@deoren
Copy link
Contributor

deoren commented Nov 16, 2018

I'll try to make some time to test this and provide feedback.

@rgerhards

Provided I didn't go anything odd, it appears that the fromhost-ip value is still not being set.. I used the config file here:

deoren/rsyslog-examples@cfd6724

and the 8.39.0 stable version from the Adiscon PPA to test.

I'm attaching debug log and apt history log files generated by the referenced rsyslog configuration.

rsyslog-debug-and-apt-history-logs-2018-11-15.zip

@zhangying451335937
Copy link

rsyslog的fromhost-ip 测试本机为空,拿不到,请教一下,可以解答吗

@davidelang
Copy link
Contributor

davidelang commented Jun 24, 2019 via email

@zhangying451335937
Copy link

zhangying451335937 commented Jun 24, 2019 via email

@zhangying451335937
Copy link

zhangying451335937 commented Jun 24, 2019

nginx template

template(name="nginxAccessTemplate" type="list" ){
constant(value="demo1") #自定义常量
constant(value=", ") #分隔符
property(name="hostname") #集群IP别名
constant(value=", ") #分隔符
property(name="syslogtag") #日志名称
constant(value=", ") #分隔符
constant(value="日志内容") #自定义常量
constant(value=", ") #分隔符
property(name="msg") #转发日志内容
constant(value="', ")
property(name="fromhost-ip")

}
在这些官方支持的配置中,hostname,syslogtag,msg都可以拿到,但是fromhost-ip却拿不到,请教一下为什么

Edit: from google translate:
In these officially supported configurations, hostname, syslogtag, msg can be obtained, but fromhost-ip can not get, ask why

@rgerhards
Copy link
Member

what should fromhost-ip be? 127.0.0.1? would that be useful to anyone? Remember that a system can have many IP addresses. since the message wasn't received from another system, there isn't an obviously right value to put in fromhost-ip

Nevertheless, the property must always be present. We use the loopback interface for that (currently only 127.0.0.1, need to consider IPv6-only systems... in the future). In my testing, I always found the property populated.

@HemanthStark
Copy link

@elain Before the problem is fixed, it can be a temporary solution to use a placeholder in template file, and replace it with external IP by start script.

external_ip=$(echo $(/sbin/ip route get 1.1.1.1 | sed -n 's/.* src //p'))

Taught by @ymattw :-)

Hi, Can you please expand the solution I am trying to forward syslog via imfile and fromhost-ip parameter in template showing blank at server , what you mean by replace external_ip by start script?

@davidelang
Copy link
Contributor

davidelang commented Sep 20, 2023 via email

@HemanthStark
Copy link

Hi @davidelang I am trying to forward eth0 iPaddress to the server via imfile , without imfile it is working fine, I am able to forward the eth0 ipaddress but with imfile it is not working.

@davidelang
Copy link
Contributor

davidelang commented Feb 13, 2024 via email

@HemanthStark
Copy link

yeah @davidelang I am trying to find a way to forward eth0 IP-address instead of loopback address to rsyslog server via imfile
using below template

module(load="imfile")
input(type="imfile" file="/var/Test.log" Tag="Testing" Severity="notice")
$template RFC5424Format,"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %FROMHOST-IP% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg% \n"

@davidelang
Copy link
Contributor

davidelang commented Feb 14, 2024 via email

@HemanthStark
Copy link

Thanks a lot @davidelang for the response.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests