Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Plugin tdengine sink not found when run ekuiper from docker #2314

Closed
edgego opened this issue Oct 13, 2023 · 11 comments
Closed

Plugin tdengine sink not found when run ekuiper from docker #2314

edgego opened this issue Oct 13, 2023 · 11 comments
Labels
help wanted Extra attention is needed
Milestone
1.14

Comments

@edgego
Copy link

edgego commented Oct 13, 2023 •
edited
Loading

Dear team,
built plugin tdengine into docker , when create a tdengine rule , but reported: sink tdengine not found. Follow Dockerfile:
Please help check the issue:

FROM golang:1.20.8 AS builder
ENV GOPROXY=https://goproxy.cn
ARG TARGETARCH

RUN sed -e 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' -i~ /etc/apk/repositories
RUN apk add --update --no-cache bash zip zeromq-dev make git gcc g++ pkgconfig libc-dev musl-dev libsodium-dev libgcc libstdc++

ADD . /go/kuiper

WORKDIR /go/kuiper

RUN make tidy
RUN make build_with_edgex
RUN make plugins

FROM alpine:edge

RUN sed -e 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' -i~ /etc/apk/repositories
RUN apk add --update --no-cache zeromq

ENV MAINTAINER="EdgeGo"
KUIPER_HOME="/kuiper"
KUIPER__BASIC__CONSOLELOG=true

ARG KUIPER_USER="kuiper"
ARG KUIPER_USER_ID="1001"

WORKDIR ${KUIPER_HOME}

RUN adduser -DH -s /sbin/nologin -u ${KUIPER_USER_ID} ${KUIPER_USER} &&
chown -Rh ${KUIPER_USER}:${KUIPER_USER} ${KUIPER_HOME} &&
mkdir -p /usr/local/taos &&
chown -Rh ${KUIPER_USER}:${KUIPER_USER} /usr/local/taos &&
mkdir -p /var/log/taos &&
chmod 777 /var/log/taos &&
chown -Rh ${KUIPER_USER}:${KUIPER_USER} /var/log/taos

COPY --chown=${KUIPER_USER}:${KUIPER_USER} --from=builder /usr/local/taos /usr/local/taos
COPY --chown=${KUIPER_USER}:${KUIPER_USER} --from=builder /usr/local/taos/include/tao*.h /usr/include/

RUN ln -s /usr/local/taos/driver/libtaos.so.1 /usr/lib/libtaos.so
RUN if [ -d /usr/lib64 ]; then
ln -s /usr/local/taos/driver/libtaos.so.1 /usr/lib64/libtaos.so ;
fi

USER ${KUIPER_USER}

RUN ldconfig /usr/lib /usr/lib64 /usr/local/lib

COPY --chown=${KUIPER_USER}:${KUIPER_USER} ./deploy/docker/docker-entrypoint.sh /usr/bin/docker-entrypoint.sh
COPY --chown=${KUIPER_USER}:${KUIPER_USER} --from=builder /go/kuiper/_build/kuiper-* /kuiper/
COPY --chown=${KUIPER_USER}:${KUIPER_USER} --from=builder /go/kuiper/extensions/sinks/ /kuiper/plugins/sinks/

RUN ln -s /kuiper/plugins/sinks/image/image@1.11.2.so /kuiper/plugins/sinks/image.so
RUN ln -s /kuiper/plugins/sinks/influx/influx@1.11.2.so /kuiper/plugins/sinks/influx.so
RUN ln -s /kuiper/plugins/sinks/influx2/influx2@1.11.2.so /kuiper/plugins/sinks/influx2.so
RUN ln -s /kuiper/plugins/sinks/kafka/kafka@1.11.2.so /kuiper/plugins/sinks/kafka.so
RUN ln -s /kuiper/plugins/sinks/sql/sql@1.11.2.so /kuiper/plugins/sinks/sql.so
RUN ln -s /kuiper/plugins/sinks/tdengine/tdengine@1.11.2.so /kuiper/plugins/sinks/tdengine.so
RUN ln -s /kuiper/plugins/sinks/zmq/zmq@1.11.2.so /kuiper/plugins/sinks/zmq.so

VOLUME ["${KUIPER_HOME}/etc", "${KUIPER_HOME}/data", "${KUIPER_HOME}/plugins", "${KUIPER_HOME}/log"]
EXPOSE 9081 20498

ENTRYPOINT ["/usr/bin/docker-entrypoint.sh"]

CMD ["./bin/kuiperd"]
@ngjaying
Copy link
Collaborator

Tdengine client seems not compatible with alpine so that we don't provide pre-built plugin. Is it possible to use -slim image and just install the prebuilt version?

@edgego
Copy link
Author

edgego commented Oct 13, 2023

@ngjaying Thanks for replay , if want compile Tdengine client inside docker , is there source code for libtaos.so?

@ngjaying
Copy link
Collaborator

@ngjaying Thanks for replay , if want compile Tdengine client inside docker , is there source code for libtaos.so?

You'll need to check with tdengine. https://github.com/taosdata/driver-go. If you can get it work, please let us know. Thanks.

@edgego
Copy link
Author

edgego commented Oct 14, 2023 •
edited
Loading

RUN git clone https://github.com/taosdata/TDengine.git -b ver-3.1.1.4
RUN cd TDengine
&& ./build.sh
&& cd debug && make install/local
After make install/local , write output to /usr/local/taos,

Also update build-plugins.sh, comment download :
...
tdengine )
#if [ "$(uname -m)" = "x86_64" ]; then
# wget "https://www.taosdata.com/assets-download/TDengine-client-2.4.0.18-Linux-x64.tar.gz" -O /tmp/TDengine-client-2.4.0.18.tar.gz;
#fi;
#if [ "$(uname -m)" = "aarch64" ]; then
# wget "https://www.taosdata.com/assets-download/TDengine-client-2.4.0.18-Linux-aarch64.tar.gz" -O /tmp/TDengine-client-2.4.0.18.tar.gz;
#fi;

        #tar -zxvf /tmp/TDengine-client-2.4.0.18.tar.gz
        #cd TDengine-client-2.4.0.18 && rm install_client.sh && cp ../install_client.sh . &&  ./install_client.sh  && cd -
        go build -trimpath --buildmode=plugin -tags plugins -o extensions/$PLUGIN_TYPE/$PLUGIN_NAME/$PLUGIN_NAME@$VERSION.so extensions/$PLUGIN_TYPE/$PLUGIN_NAME/$PLUGIN_NAME.go

@ngjaying
Copy link
Collaborator

RUN git clone https://github.com/taosdata/TDengine.git -b ver-3.1.1.4 RUN cd TDengine && ./build.sh && cd debug && make install/local After make install/local , write output to /usr/local/taos,

Also update build-plugins.sh, comment download : ... tdengine ) #if [ "$(uname -m)" = "x86_64" ]; then # wget "https://www.taosdata.com/assets-download/TDengine-client-2.4.0.18-Linux-x64.tar.gz" -O /tmp/TDengine-client-2.4.0.18.tar.gz; #fi; #if [ "$(uname -m)" = "aarch64" ]; then # wget "https://www.taosdata.com/assets-download/TDengine-client-2.4.0.18-Linux-aarch64.tar.gz" -O /tmp/TDengine-client-2.4.0.18.tar.gz; #fi;

        #tar -zxvf /tmp/TDengine-client-2.4.0.18.tar.gz
        #cd TDengine-client-2.4.0.18 && rm install_client.sh && cp ../install_client.sh . &&  ./install_client.sh  && cd -
        go build -trimpath --buildmode=plugin -tags plugins -o extensions/$PLUGIN_TYPE/$PLUGIN_NAME/$PLUGIN_NAME@$VERSION.so extensions/$PLUGIN_TYPE/$PLUGIN_NAME/$PLUGIN_NAME.go

Do you mean this can build tdengine for alpine? Would you please help to create a basic PR？

@edgego
Copy link
Author

edgego commented Oct 16, 2023

not alpine, use ubuntu, from tdengine github just use ubuntu to build tdengine.

@edgego
Copy link
Author

edgego commented Oct 19, 2023

TDengine can be compiled for alpine, tested version 3.1.1.4 and 3.2.0.0

@edgego
Copy link
Author

edgego commented Nov 17, 2023

check image edgego/ekuiper:v1.11.5, including plugin tdengine

@ngjaying ngjaying added the help wanted Extra attention is needed label Dec 27, 2023
@ngjaying
Copy link
Collaborator

Need help to update build_plugins.sh to build tdengine plugin for alpine.

@ngjaying ngjaying mentioned this issue Jan 15, 2024
Closed
@edgego
Copy link
Author

edgego commented Jan 15, 2024 •
edited
Loading

Dowload file, that include Dockerfile and build_plugins.sh
ekuiper-tdengine.txt

Suggest download TDengine.git to local , and update libuv_CMakeLists.txt.in , GIT_TAG from v1.44.2 to v1.46.0, can refer taosdata/TDengine#23325

Also update go.mod from ekuiper dir extensions github.com/taosdata/driver-go/v3 v3.5.0

@ngjaying
Copy link
Collaborator

Thank you @edgego . I prefer to have this automated so that we can build the alpine version. This issue is still open to pursuit this goal.

@ngjaying ngjaying added this to the 1.14 milestone May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Extra attention is needed
Projects
None yet
Milestone
1.14
Development

No branches or pull requests
2 participants
@ngjaying @edgego