Parquet files on /tmp/nanomq-parquet not available after config MQTT Messaging Queue

[canob]

Hi everybody,

Can anybody answer me if I'm understanding this correctly?

I configured MQTT Messaging Queue to save parquet files on /tmp/nanomq-parquet, as explained on https://nanomq.io/docs/en/latest/config-description/parquet.html

My added lines to nanomq.conf:

exchange_client.mq1 {
        exchange {
                topic = "winevt",
                name = "winevt",
                ringbus = {
                        name = "ringbus",
                        cap = 10,
                        fullOp = 2
                }
        }
}
parquet {
        compress = uncompressed
        encryption {
                key_id = kf
                key = "0123456789012345"
                type = AES_GCM_V1
        }
        dir = "/tmp/nanomq-parquet"
        file_name_prefix = "parquet"
        file_count = 50
        file_size = 10KB
}

After configured debug level logs on nanomq.log, I see these logs in the file:

2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/pub_handler.c:1540 decode_pub_message: cmd: 3, retain: 0, qos: 0, dup: 0, remaining length: 1676
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/pub_handler.c:1588 decode_pub_message: topic: [winevt], len: [6], qos: 0
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/pub_handler.c:1633 decode_pub_message: used pos: [8]
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/pub_handler.c:1645 decode_pub_message: payload: [{"ProviderName":"Microsoft-Windows-Security-Auditing","ProviderGUID":"","EventID":"4624","Level":"0","Task":"12544","Opcode":"0","Keywords":"0x8020000000000000","TimeCreated":"2024/03/04 18:31:44.861126700","EventRecordID":"43491076","ActivityID":"","RelatedActivityID":"","ProcessID":"932","ThreadID":"8860","Channel":"Security","Computer":"XXXXXXXXXXX","UserID":"","Version":"2","DescriptionTitle":"An account was successfully logged on.","subject.security_id":"S-1-0-0","subject.account_name":"-","subject.account_domain":"-","subject.logon_id":"0x0","logon_information.logon_type":"3","logon_information.restricted_admin_mode":"-","logon_information.virtual_account":"No","logon_information.elevated_token":"Yes","impersonation_level":"Impersonation","new_logon.security_id":"S-1-5-21-1844237615-1563985344-1957994488-21392","new_logon.account_name":"XXXXXXXXXXXXX","new_logon.account_domain":"XXXXXXXXX","new_logon.logon_id":"0x4275264C","new_logon.linked_logon_id":"0x0","new_logon.network_account_name":"-","new_logon.network_account_domain":"-","new_logon.logon_guid":"{7241b8a7-7980-773b-6042-1e70a2d6e81c}","process_information.process_id":"0x0","process_information.process_name":"-","network_information.workstation_name":"-","network_information.source_network_address":"10.11.0.130","network_information.source_port":"15602","detailed_authentication_information.logon_process":"Kerberos","detailed_authentication_information.authentication_package":"Kerberos","detailed_authentication_information.transited_services":"-","detailed_authentication_information.package_name_(ntlm_only)":"-","detailed_authentication_information.key_length":"0"}], len = 1668
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/pub_handler.c:1190 handle_pub: pipe_info size: [0]
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nng/src/mqtt/protocol/exchange/exchange.c:130 exchange_handle_msg: msg enqueued! msg 0x7fd02400bb20 key: 1688924799
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/apps/broker.c:548 server_cb: WAIT ^^^^ ctx4 ^^^^
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/apps/broker.c:627 server_cb: SEND ^^^^ ctx4 ^^^^
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/pub_handler.c:1286 free_pub_packet: free topic
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/pub_handler.c:1302 free_pub_packet: free payload
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/pub_handler.c:1308 free_pub_packet: free pub_packet
2024-03-04 18:31:21 [13] DEBUG /home/runner/work/nanomq/nanomq/nanomq/apps/broker.c:300 server_cb: RECV  ^^^^ ctx3 ^^^^

So, at my understanding, the queue is working:
/home/runner/work/nanomq/nanomq/nng/src/mqtt/protocol/exchange/exchange.c:130 exchange_handle_msg: msg enqueued! msg XXXXXXXXXX key: XXXXXXXXX

But I cannot see any .parquet files on /tmp/nanomq-parquet folder.

Anyone can explain me what I'm doing wrong or what I'm not understanding?

Thanks in advance for your help.

[canob]

Additionally, I'm seeing this error message in the logs:
2024-03-05 12:08:45 [1] ERROR /home/runner/work/nanomq/nanomq/nanomq/apps/broker.c:1095 broker: Exchange url is not set

In nanomq_example.conf example file, you can see a exchange_url parameter configured under exchange_client.mql:

exchange_client.mq1 {
        exchange_url = "tcp://127.0.0.1:10000"
        exchange {
                topic = "exchange/topic1",
                name = "exchange_no1",
                ringbus = {
                        name = "ringbus",
                        cap = 1000,
                        fullOp = 2
                }

But here in any place say anything about exchange_url:
https://nanomq.io/docs/en/latest/config-description/parquet.html#configuration-items

[JaylinYu]

The exchange url is the url for you to consume msgs in MQ via NNG REQ/REP. Ref: https://nanomq.io/docs/en/latest/mqtt-stream/consumer.html

I tried your config, it works fine (you will see logs like "flush parquet to disk ... "). You might be using a version that MQTT Stream (parquet) is embedded. currently, the binary releases are not parquet enabled by default. Plz compile from the source or wait for the official binary release.

[canob]

Thanks for your answer @JaylinYu . Is there any roadmap of when the official binary release (I'm using "full" docker image) is going to include parquet enabled by default?
I tried to compile from source and update the binaries in the docker image that I'm using, but with no luck (I cannot run nanomq inside the docker container after the update because many needed libraries are not available)

[canob]

Hi @JaylinYu,
I finally compiled from source successfully nanomq, and updated the binary on the docker image, but still not seeing any parquet files on /tmp/nanomq-parquet, :(
Any other thing that I can review to understand what is happening?
Thanks in advance.

[JaylinYu]

Hi @canob Alejandro , sorry didnt make it easier for you.

Need following options to be enabled while compiling from source: -DENABLE_PARQUET=ON
This also require you to have arrow c++ installed first. https://github.com/apache/arrow

[canob]

Installed Apache Arrow C++ using apt on Ubuntu based on this, https://arrow.apache.org/install/, but after that, trying to compile with Ninja like before, adding now -DENABLE_PARQUET=ON, I got many errors, beeing the main one (I think) this one:

[267/546] Building CXX object nng/CMakeFiles/nng.dir/src/supplemental/nanolib/parquet/parquet.cc.o
FAILED: nng/CMakeFiles/nng.dir/src/supplemental/nanolib/parquet/parquet.cc.o
/usr/bin/c++  -DACL_SUPP -DENABLE_LOG -DNANO_HAVE_ABSTRACT_SOCKETS -DNANO_HIDDEN_VISIBILITY -DNANO_PLATFORM_LINUX -DNANO_PLATFORM_POSIX -DNANO_USE_EVENTFD -DNNG_ENABLE_COMPAT -DNNG_ENABLE_IPV6 -DNNG_ENABLE_STATS -DNNG_HAVE_ABSTRACT_SOCKETS -DNNG_HAVE_BACKTRACE=1 -DNNG_HAVE_BUS0 -DNNG_HAVE_CLOCK_GETTIME=1 -DNNG_HAVE_EPOLL=1 -DNNG_HAVE_EPOLL_CREATE1=1 -DNNG_HAVE_EVENTFD=1 -DNNG_HAVE_FLOCK=1 -DNNG_HAVE_GETRANDOM=1 -DNNG_HAVE_INET6=1 -DNNG_HAVE_LANGINFO=1 -DNNG_HAVE_LIBATOMIC=1 -DNNG_HAVE_LIBNSL=1 -DNNG_HAVE_LOCKF=1 -DNNG_HAVE_MQTT_BROKER -DNNG_HAVE_MQTT_CLIENT -DNNG_HAVE_MSG_CONTROL=1 -DNNG_HAVE_PAIR0 -DNNG_HAVE_PAIR1 -DNNG_HAVE_PTHREAD_ATFORK_PTHREAD=1 -DNNG_HAVE_PTHREAD_SETNAME_NP=1 -DNNG_HAVE_PUB0 -DNNG_HAVE_PULL0 -DNNG_HAVE_PUSH0 -DNNG_HAVE_REP0 -DNNG_HAVE_REQ0 -DNNG_HAVE_RESPONDENT0 -DNNG_HAVE_SEMAPHORE_PTHREAD=1 -DNNG_HAVE_SOCKETPAIR=1 -DNNG_HAVE_SOPEERCRED=1 -DNNG_HAVE_STDATOMIC=1 -DNNG_HAVE_STRCASECMP=1 -DNNG_HAVE_STRNCASECMP=1 -DNNG_HAVE_STRNLEN=1 -DNNG_HAVE_SUB0 -DNNG_HAVE_SURVEYOR0 -DNNG_HAVE_UNIX_SOCKETS=1 -DNNG_HIDDEN_VISIBILITY -DNNG_MAX_EXPIRE_THREADS=8 -DNNG_MAX_POLLER_THREADS=8 -DNNG_MAX_TASKQ_THREADS=16 -DNNG_PLATFORM_LINUX -DNNG_PLATFORM_POSIX -DNNG_PRIVATE -DNNG_RESOLV_CONCURRENCY=4 -DNNG_STATIC_LIB -DNNG_SUPP_HTTP -DNNG_TRANSPORT_FDC -DNNG_TRANSPORT_INPROC -DNNG_TRANSPORT_IPC -DNNG_TRANSPORT_MQTT_BROKER_TCP -DNNG_TRANSPORT_MQTT_BROKER_TLS -DNNG_TRANSPORT_MQTT_BROKER_WS -DNNG_TRANSPORT_MQTT_TCP -DNNG_TRANSPORT_MQTT_TLS -DNNG_TRANSPORT_TCP -DNNG_TRANSPORT_TLS -DNNG_TRANSPORT_WS -DNNG_USE_EVENTFD -DSUPP_CLIENT -DSUPP_NNG_PROXY -DSUPP_PARQUET -DSUPP_SYSLOG -D_GNU_SOURCE -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -D_THREAD_SAFE -I../nng/src -I../nng/include -I../nanolib/include -I../nanomq/include -Wall -Wextra -fno-omit-frame-pointer     -std=gnu++11 -MD -MT nng/CMakeFiles/nng.dir/src/supplemental/nanolib/parquet/parquet.cc.o -MF nng/CMakeFiles/nng.dir/src/supplemental/nanolib/parquet/parquet.cc.o.d -o nng/CMakeFiles/nng.dir/src/supplemental/nanolib/parquet/parquet.cc.o -c ../nng/src/supplemental/nanolib/parquet/parquet.cc

I think that I'm going to wait for the binary release with parquet support, :(
Thanks anyway @JaylinYu