使用docker构建，每次使用adapter增删改数据到es，内存都会上涨，最后爆掉 #4836
Comments
|
我对docker canal-adapter内部的start.sh进行了修改，似乎好像解决了这个问题？但是只是本地部署没有看出问题，不知道线上如何。先记录一下修改方案 以下修改方案基于 首先docker exec进入到容器内部，修改启动脚本 我修改了两个地方
#!/bin/bash
current_path=`pwd`
case "`uname`" in
Linux)
bin_abs_path=$(readlink -f $(dirname $0))
;;
*)
bin_abs_path=`cd $(dirname $0); pwd`
;;
esac
base=${bin_abs_path}/..
export LANG=en_US.UTF-8
export BASE=$base
if [ -f $base/bin/adapter.pid ] ; then
echo "found adapter.pid , Please run stop.sh first ,then startup.sh" 2>&2
exit 1
fi
if [ ! -d $base/logs ] ; then
mkdir -p $base/logs
fi
## set java path
if [ -z "$JAVA" ] ; then
JAVA=$(which java)
fi
ALIBABA_JAVA="/usr/alibaba/java/bin/java"
TAOBAO_JAVA="/opt/taobao/java/bin/java"
if [ -z "$JAVA" ]; then
if [ -f $ALIBABA_JAVA ] ; then
JAVA=$ALIBABA_JAVA
elif [ -f $TAOBAO_JAVA ] ; then
JAVA=$TAOBAO_JAVA
else
echo "Cannot find a Java JDK. Please set either set JAVA or put java (>=1.5) in your PATH." 2>&2
exit 1
fi
fi
case "$#"
in
0 )
;;
2 )
if [ "$1" = "debug" ]; then
DEBUG_PORT=$2
DEBUG_SUSPEND="n"
JAVA_DEBUG_OPT="-Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=$DEBUG_PORT,server=y,suspend=$DEBUG_SUSPEND"
fi
;;
* )
echo "THE PARAMETERS MUST BE TWO OR LESS.PLEASE CHECK AGAIN."
exit;;
esac
str=`file -L $JAVA | grep 64-bit`
if [ -n "$str" ]; then
JAVA_OPTS="-server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -Xss256k -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError"
else
JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxPermSize=128m "
fi
## 主要是修改了这个地方，修改了启动参数
JAVA_OPTS="-server -Xms104m -Xmx512m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:MaxDirectMemorySize=128m -XX:MaxPermSize=128m "
echo $JAVA_OPTS
JAVA_OPTS=" $JAVA_OPTS -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8"
ADAPTER_OPTS="-DappName=canal-adapter"
for i in $base/lib/*;
do CLASSPATH=$i:"$CLASSPATH";
done
CLASSPATH="$base/conf:$CLASSPATH";
echo "cd to $bin_abs_path for workaround relative path"
cd $bin_abs_path
echo CLASSPATH :$CLASSPATH
exec $JAVA $JAVA_OPTS $JAVA_DEBUG_OPT $ADAPTER_OPTS -classpath .:$CLASSPATH com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication插入前 插入中 插入后 可以看到插入几万条数据后（确实也同步到ES中了），adapter的内存反而还下降了？不是很懂，只能说回想起了深度学习调参的日子 |
|
m |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
环境
ubuntu 22.04
mysql
canal 1.1.4
canal-adapter 1.1.4/1.1.5（均试过，均会出现内存上涨）
elasticsearch 7.8
以上环境均使用docker安装
docker pull elasticsearch:7.8.0 docker pull kibana:7.8.0 docker pull canal/canal-server:v1.1.4 # 以下三种全都试了，似乎都会有问题 docker pull slpcat/canal-adapter:v1.1.5-jdk8 docker pull slpcat/canal-adapter:v1.1.4 docker pull slpcat/canal-adapter:v1.1.5问题描述
使用docker监控内存命令（docker stats）发现，每次从MySQL到elasticsearch同步数据时，canal-adapter占用内存均会上涨，并且无法回落，导致内存满了kill掉该容器。
同步的配置文件
其中News_4的数据量有4w条
全量同步的命令
内存监控
下为图1 刚启动
下为图2 插入数据一次
下为图3 boom，爆炸
监控界面直接空白，过一段时间后会自动杀掉该容器。
命令行输出：
其他canal的配置
canal.properties
canal.properties
canal-adapter的application.yml
server: port: 8081 spring: jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts: syncBatchSize: 1000 retries: 0 timeout: accessKey: secretKey: consumerProperties: # canal tcp consumer canal.tcp.server.host: canal:11111 #canal.tcp.zookeeper.hosts: canal.tcp.batch.size: 500 canal.tcp.username: canal.tcp.password: # kafka consumer # kafka.bootstrap.servers: 127.0.0.1:9092 # kafka.enable.auto.commit: false # kafka.auto.commit.interval.ms: 1000 # kafka.auto.offset.reset: latest # kafka.request.timeout.ms: 40000 # kafka.session.timeout.ms: 30000 # kafka.isolation.level: read_committed # kafka.max.poll.records: 1000 # # rocketMQ consumer # rocketmq.namespace: # rocketmq.namesrv.addr: 127.0.0.1:9876 # rocketmq.batch.size: 1000 # rocketmq.enable.message.trace: false # rocketmq.customized.trace.topic: # rocketmq.access.channel: # rocketmq.subscribe.filter: # # rabbitMQ consumer # rabbitmq.host: # rabbitmq.virtual.host: # rabbitmq.username: # rabbitmq.password: # rabbitmq.resource.ownerId: srcDataSources: defaultDS: url: jdbc:mysql://mysql:3306/MyTest?useUnicode=true&characterEncoding=UTF-8&useSSL=false username: root password: {我的密码} maxActive: 2 canalAdapters: - instance: example # canal instance Name or mq topic name groups: - groupId: g1 outerAdapters: - name: logger # - name: rdb # key: mysql1 # properties: # jdbc.driverClassName: com.mysql.jdbc.Driver # jdbc.url: jdbc:mysql://127.0.0.1:3306/mytest2?useUnicode=true # jdbc.username: root # jdbc.password: 121212 # - name: rdb # key: oracle1 # properties: # jdbc.driverClassName: oracle.jdbc.OracleDriver # jdbc.url: jdbc:oracle:thin:@localhost:49161:XE # jdbc.username: mytest # jdbc.password: m121212 # - name: rdb # key: postgres1 # properties: # jdbc.driverClassName: org.postgresql.Driver # jdbc.url: jdbc:postgresql://localhost:5432/postgres # jdbc.username: postgres # jdbc.password: 121212 # threads: 1 # commitSize: 3000 # - name: hbase # properties: # hbase.zookeeper.quorum: 127.0.0.1 # hbase.zookeeper.property.clientPort: 2181 # zookeeper.znode.parent: /hbase # - name: es6 # hosts: elasticsearch:9200 # 127.0.0.1:9200 for rest mode # properties: # mode: rest # or rest # # security.auth: test:123456 # only used for rest mode # cluster.name: elasticsearch - name: es7 hosts: elasticsearch_7_8:9200 # 127.0.0.1:9200 for rest mode properties: mode: rest # or rest # security.auth: test:123456 # only used for rest mode cluster.name: elasticsearch # - name: kudu # key: kudu # properties: # kudu.master.address: 127.0.0.1 # ',' split multi address不知道有没有人遇到类似的情况？如果有需要补充详细信息，请指出
The text was updated successfully, but these errors were encountered: