Why the result is not good on the Dataset——2018CCFBDCI汽车行业用户观点主题及情感识别？

[pengwei-iie]

Prerequisites

Please fill in by replacing [ ] with [x].

• Are you running the latest bert-as-service?
• Did you follow the installation and the usage instructions in README.md?
• Did you check the FAQ list in README.md?
• Did you perform a cursory search on existing issues?

System information

Some of this information can be collected via this script.

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• TensorFlow installed from (source or binary):source
• TensorFlow version:1.12.0
• Python version:3.6.7
• bert-as-service version: the former version，not the lastest
• GPU model and memory:10G
• CPU model and memory:

---

Description

Please replace YOUR_SERVER_ARGS and YOUR_CLIENT_ARGS accordingly. You can also write your own description for reproducing the issue.

I'm using this command to start the server:

python app.py -pooling_strategy NONE -model_dir ./chinese_L-12_H-768_A-12/ -num_worker=1

and calling the server via:

bc = BertClient()
x_= bc.encode(x_batch) # (batch_size, seq_len) and get x_(batch_size, seq_len, 768)

Then this issue shows up:
I have used the sentence embedding to the model—— text classification, just connected a DNN and a classifier to predict the label(such as price, power, space, security and so on) base on comment on cars.
But the result is very bad.
Then I used the word embedding to the RNN and softmax,is bad too.I do not know why.
...

[pengwei-iie]

I have used the sentence embedding to the model—— text classification, just connected a DNN and a classifier to predict the label(such as price, power, space, security and so on) base on comment on cars.
But the result is very bad.
Then I used the word embedding to the RNN and softmax,is bad too.I do not know why.

[astariul]

Do you have an implementation that works correctly ?

[pengwei-iie]

Do you have an implementation that works correctly ?

Yes, I used the same model(except the word embedding that I initialized it randomly) and had a not bad results whose accuracy is 70%,however, the bert in this github is only 24%.
The word embedding in my code about bert is as follow：
`

for epoch in range(config.num_epochs):  
    batch_train = batch_iter(x_train, y_train, config.batch_size)
    for x_batch, y_batch in batch_train:
        x_batch = bc.encode(x_batch)
        feed_dict = feed_data(x_batch, y_batch, config.dropout_keep_prob)
        session.run(model.optim, feed_dict=feed_dict)  # 运行优化
        total_batch += 1`

Here， the x_batch is（128，80，768）

The total code of my model is：
`
#!/usr/bin/python

import tensorflow as tf
from data.cnews_loader import attention

from service.client import BertClient

class TRNNConfig(object):
"""RNN配置参数"""

# 模型参数
embedding_dim = 768      # 词向量维度
seq_length = 80        # 序列长度
num_classes = 10        # 类别数
vocab_size = 5000       # 词汇表达小

num_layers= 2           # 隐藏层层数
hidden_dim = 128        # 隐藏层神经元
rnn = 'gru'             # lstm 或 gru

attention_dim = 50
l2_reg_lambda = 0.01

dropout_keep_prob = 1.0 # dropout保留比例
learning_rate = 1e-3    # 学习率

batch_size = 128         # 每批训练大小
num_epochs = 20          # 总迭代轮次

print_per_batch = 100    # 每多少轮输出一次结果
save_per_batch = 20      # 每多少轮存入tensorboard

class TextRNN(object):
"""文本分类，RNN模型"""
def init(self, config):
self.config = config

    # 三个待输入的数据
    # self.input_x = tf.placeholder(tf.int32, [None, self.config.seq_length], name='input_x')
    self.input_x = tf.placeholder(tf.float32, [None, self.config.seq_length, self.config.embedding_dim], name='input_x')
    self.input_y = tf.placeholder(tf.float32, [None, self.config.num_classes], name='input_y')
    self.keep_prob = tf.placeholder(tf.float32, name='keep_prob')

    self.rnn()

def rnn(self):
    """rnn模型"""

    def lstm_cell():   # lstm核
        return tf.contrib.rnn.BasicLSTMCell(self.config.hidden_dim, state_is_tuple=True)

    def gru_cell():  # gru核
        return tf.contrib.rnn.GRUCell(self.config.hidden_dim)

    def dropout(): # 为每一个rnn核后面加一个dropout层
        if (self.config.rnn == 'lstm'):
            cell = lstm_cell()
        else:
            cell = gru_cell()
        return tf.contrib.rnn.DropoutWrapper(cell, output_keep_prob=self.keep_prob)

    # 词向量映射
    # with tf.device('/cpu:0'):
    #     embedding = tf.get_variable('embedding', [self.config.vocab_size, self.config.embedding_dim])
    #     embedding_inputs = tf.nn.embedding_lookup(embedding, self.input_x)

    with tf.name_scope("rnn"):
        # 多层rnn网络
        cells = [dropout() for _ in range(self.config.num_layers)]
        # cell = dropout()
        rnn_cell = tf.contrib.rnn.MultiRNNCell(cells, state_is_tuple=True)
        # (batch_size, num_step, embeddings)  (b, 80, 128)
        _outputs, _ = tf.nn.dynamic_rnn(cell=rnn_cell, inputs=self.input_x, dtype=tf.float32)

        # new
        # output = attention(_outputs, self.config.attention_dim, self.config.l2_reg_lambda)
        # (b, 128)
        last = _outputs[:, -1, :]  # 取最后一个时序输出作为结果(b, 128)

    with tf.name_scope("score"):
        # bc = BertClient()
        # self.input (128)
        # output = bc.encode(self.input_x)  # (batch_size, 768)
        # 全连接层，后面接dropout以及relu激活
        fc = tf.layers.dense(last, self.config.hidden_dim, name='fc1')
        fc = tf.contrib.layers.dropout(fc, self.keep_prob)
        fc = tf.nn.relu(fc)
        # fc = tf.layers.dense(fc, 512, name='fc2')
        # fc = tf.contrib.layers.dropout(fc, self.keep_prob)
        # fc = tf.nn.relu(fc)
        #
        # fc = tf.layers.dense(fc, 256, name='fc3')
        # fc = tf.contrib.layers.dropout(fc, self.keep_prob)
        # fc = tf.nn.relu(fc)
        # fc = tf.layers.dense(fc, 128, name='fc4')
        # fc = tf.contrib.layers.dropout(fc, self.keep_prob)
        # fc = tf.nn.relu(fc)

        # 分类器(b, 128)->(b, 10)
        self.logits = tf.layers.dense(fc, self.config.num_classes, name='fc5')
        self.y_pred_cls = tf.argmax(tf.nn.softmax(self.logits), 1)  # 预测类别

    with tf.name_scope("optimize"):
        # 损失函数，交叉熵
        cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=self.logits, labels=self.input_y)
        self.loss = tf.reduce_mean(cross_entropy)
        # 优化器
        self.optim = tf.train.AdamOptimizer(learning_rate=self.config.learning_rate).minimize(self.loss)

    with tf.name_scope("accuracy"):
        # 准确率
        correct_pred = tf.equal(tf.argmax(self.input_y, 1), self.y_pred_cls)
        self.acc = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

`

[hanxiao]

could you paste the command for starting the server?

[pengwei-iie]

could you paste the command for starting the server?

python app.py -pooling_strategy NONE -model_dir ./chinese_L-12_H-768_A-12/ -num_worker=1

[hanxiao]

could you fill in the issue form so that I can know which version of bert-as-service are you using?

[pengwei-iie]

could you fill in the issue form so that I can know which version of bert-as-service are you using?

OK，I have update the issue form.
If I fine tune the model,will it take a long time?
Thank you!

[hanxiao]

Before changing your model or doing fine-tuning, I strongly suggest you to use the latest bert-as-service and give a retry. Please do

pip install -U bert-serving-server bert-serving-client

Based on your server command and the form, I suspect you are using a pretty old version, which was even not installed via pip. Note that, there was some bugs in previous version, and was fixed since 1.5.0. These bugs may degrade the accuracy.

[pengwei-iie]

Before changing your model or doing fine-tuning, I strongly suggest you to use the latest bert-as-service and give a retry. Please do

pip install -U bert-serving-server bert-serving-client

Based on your server command and the form, I suspect you are using a pretty old version, which was even not installed via pip. Note that, there was some bugs in previous version, and was fixed since 1.5.0. These bugs may degrade the accuracy.

OK,thanks.I will try it.

[pengwei-iie]

commond：
bert-serving-start -model_dir ./chinese_L-12_H-768_A-12/ -max_seq_len 128 -port 5557 -port_out 5558 -pooling_strategy REDUCE_MEAN -num_worker=1

but the package info：
(bert) pengwei@ubuntu:~/bert-as-service-master$ conda list

#packages in environment at /home/pengwei/.conda/envs/bert:
#Name Version Build Channel
_tflow_select 2.3.0 mkl
absl-py 0.6.1 py36_0
astor 0.7.1 py36_0
bert-serving-client 1.6.2
bert-serving-server 1.6.2
blas 1.0 mkl

the return is：
AttributeError: version mismatch! server version is 1.5.7 but client version is 1.6.2!
consider "pip install -U bert-serving-server bert-serving-client"
or disable version-check by "BertClient(check_version=False)"

[hanxiao]

check your BertClient port, apparently you are connecting to an old (and living!) 1.5.7 server.