GitHub - China-ChallengeHub/Cellphone-Behavior: “2020创青春·交子杯” 新网银行金融科技挑战赛 AI算法赛道

2020创青春·交子杯” 新网银行金融科技挑战赛 AI算法赛道 TOP2

模型

CNN

1.Baseline 0.711左右的分数

https://mp.weixin.qq.com/s/r7Ai8FVSPRB71PVghYk75A 更换优化算法 0.7296349206349205

model.compile(loss='categorical_crossentropy',
                  optimizer='rmsprop',
                  metrics=['acc'])

线上0.7296349206349205

开源Baseline ot_code代码 线上0.74

23/23 [==============================] - 1s 43ms/step - loss: 0.1183 - acc: 0.9578 - val_loss: 1.1392 - val_acc: 0.7455 - lr: 7.8125e-06
Epoch 178/500
23/23 [==============================] - 1s 43ms/step - loss: 0.1159 - acc: 0.9597 - val_loss: 1.1376 - val_acc: 0.7476 - lr: 7.8125e-06
Epoch 00178: early stopping

线上0.7456984126984126

将优化算法调整为rmsprop线上分数0.7463015873015874

Epoch 161/500
23/23 [==============================] - 1s 44ms/step - loss: 0.1667 - acc: 0.9412 - val_loss: 0.9308 - val_acc: 0.7510 - lr: 7.8125e-06
Epoch 162/500
23/23 [==============================] - 1s 44ms/step - loss: 0.1751 - acc: 0.9403 - val_loss: 0.9285 - val_acc: 0.7510 - lr: 7.8125e-06
Epoch 00162: early stopping

从上面可以看出来rmsprop要优于Adam算法 3. bp_cnn 在卷积层之后添加批归一化

X = Conv2D(filters=128,
               kernel_size=(3, 3),
               activation='relu',
               padding='same')(X)
    X = BatchNormalization()(X)

训练效果有明显提升

Epoch 153/500
23/23 - 1s - loss: 0.0745 - acc: 0.9794 - val_loss: 0.8036 - val_acc: 0.7716 - lr: 7.8125e-06
Epoch 154/500
23/23 - 1s - loss: 0.0777 - acc: 0.9775 - val_loss: 0.8064 - val_acc: 0.7709 - lr: 7.8125e-06
Epoch 00154: early stopping

线上 0.7661587301587302

删除两个特征之后，线上0.7644

尝试不同的归一化方式

Group Normalization (TensorFlow Addons)

23/23 - 2s - loss: 0.0185 - acc: 0.9971 - val_loss: 0.8559 - val_acc: 0.8073 - lr: 1.5625e-05
Epoch 157/500
23/23 - 2s - loss: 0.0173 - acc: 0.9971 - val_loss: 0.8587 - val_acc: 0.8107 - lr: 7.8125e-06
Epoch 158/500
23/23 - 2s - loss: 0.0186 - acc: 0.9964 - val_loss: 0.8590 - val_acc: 0.8073 - lr: 7.8125e-06
Epoch 00158: early stopping

线上分数0.764825396825397

Instance Normalization (TensorFlow Addons)

23/23 - 2s - loss: 2.8817 - acc: 0.1030 - val_loss: 2.8588 - val_acc: 0.1021 - lr: 0.0010
Epoch 2/500
23/23 - 2s - loss: 2.8613 - acc: 0.0945 - val_loss: 2.8557 - val_acc: 0.1021 - lr: 0.0010
Epoch 3/500
23/23 - 2s - loss: 2.8608 - acc: 0.0965 - val_loss: 2.8554 - val_acc: 0.1021 - lr: 0.0010
Epoch 4/500
23/23 - 2s - loss: 2.8594 - acc: 0.0989 - val_loss: 2.8557 - val_acc: 0.1008 - lr: 0.0010

没有成功

Layer Normalization (TensorFlow Core)

Epoch 00112: ReduceLROnPlateau reducing learning rate to 6.25000029685907e-05.
23/23 - 2s - loss: 0.0067 - acc: 0.9995 - val_loss: 0.9610 - val_acc: 0.7970 - lr: 1.2500e-04
Epoch 113/500
23/23 - 2s - loss: 0.0052 - acc: 0.9998 - val_loss: 0.9456 - val_acc: 0.8038 - lr: 6.2500e-05
Epoch 114/500
23/23 - 2s - loss: 0.0045 - acc: 0.9995 - val_loss: 0.9422 - val_acc: 0.8018 - lr: 6.2500e-05
Epoch 00114: early stopping

线上分数0.7607777777777778

GlobalAveragePooling2D vs GlobalMaxPooling2D

Epoch 179/500
23/23 - 1s - loss: 0.0718 - acc: 0.9762 - val_loss: 0.8834 - val_acc: 0.7936 - lr: 1.9531e-06
Epoch 00179: early stopping
accuracy_score 0.7949245541838135 acc_combo 0.8232738911751242
5kflod mean acc score:0.7934734597517326
5kflod mean combo score:0.8238260577813273

GlobalAveragePooling2D 线上分数0.7686349206349207优于 GlobalMaxPooling2D

LSTM

双向的LSTM

23/23 - 1s - loss: 0.1118 - acc: 0.9609 - val_loss: 1.7949 - val_acc: 0.6543 - lr: 1.2500e-04
Epoch 00104: early stopping
accuracy_score 0.6694101508916324 acc_combo 0.7127833300672791
5kflod mean acc score:0.6626436732978505
5kflod mean combo score:0.7082775474080174

线上分数0.7023650793650793 2. LSTM-FCN

论文地址：LSTM Fully Convolutional Networks for Time Series Classification

参数1

def LSTM_FCN():
    input = Input(shape=(seq_len, fea_size), name="input_layer")
    x = LSTM(64)(input)
    x = Dropout(0.8)(x)

    # y = Permute((2, 1))(input)
    y = Conv1D(128, 8, padding='same', kernel_initializer='he_uniform')(input)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = Conv1D(256, 5, padding='same', kernel_initializer='he_uniform')(y)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = Conv1D(128, 3, padding='same', kernel_initializer='he_uniform')(y)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = GlobalAveragePooling1D()(y)

    x = concatenate([x, y])

    pred = Dense(19, activation='softmax')(x)
    model = Model([input], pred)
    return model
结果
accuracy_score 0.7887517146776406 acc_combo 0.8202038016852817
5kflod mean acc score:0.7870284342677916
5kflod mean combo score:0.818158419984462

参数2

def LSTM_FCN():
    input = Input(shape=(seq_len, fea_size), name="input_layer")
    x = LSTM(64)(input)
    x = Dropout(0.8)(x)

    # y = Permute((2, 1))(input)
    y = Conv1D(128, 5, padding='same', kernel_initializer='he_uniform')(input)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = Conv1D(256, 4, padding='same', kernel_initializer='he_uniform')(y)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = Conv1D(128, 3, padding='same', kernel_initializer='he_uniform')(y)
    y = BatchNormalization()(y)
    y = Activation('relu')(y)

    y = GlobalAveragePooling1D()(y)

    x = concatenate([x, y])

    pred = Dense(19, activation='softmax')(x)
    model = Model([input], pred)
    return model

线下： lstm_acc 0.7962136532999378 combo 0.8261682540488406

线上0.754 代码对应为lstm.py
加入class weight引起过拟合

MLSTM-FCN 代码：mlstm_fcn.py

原始generate_model

accuracy_score 0.7764060356652949 acc_combo 0.8090012411000044
acc_scores: [0.8163125428375599, 0.7930089102124743, 0.7908093278463649, 0.789437585733882, 0.7764060356652949]
combo_scores: [0.842749436992068, 0.8260060706942118, 0.822294075380494, 0.8216735253772278, 0.8090012411000044]
5kflod mean acc score:0.7931948804591152
5kflod mean combo score:0.8243448699088012

线上：0.761

输入train_lstm,train_lstm_inv,train_features

accuracy_score 0.8168724279835391 acc_combo 0.8431314912796375
acc_scores: [0.8122001370801919, 0.818368745716244, 0.8237311385459534, 0.821673525377229, 0.8168724279835391]
combo_scores: [0.8394203466170552, 0.8462090799308052, 0.8495656149977121, 0.8504474492128801, 0.8431314912796375]
5kflod mean acc score:0.8185691949406314
5kflod mean combo score:0.8457547964076181

线上0.776

Multi Input

cnn双输入（正向sequences和反向sequences）可以对应har.py代码，线上0.7726190476190478 线下har_acc0.8123967315118028_combo0.8401833538228309.csv

HAR：CNN输入与Dense输入融合

优化的地方：

目前尝试输入时间序列数据与特征数据，特征不能漫蛮力添加
CNN中间BN和Dropout稍微有调整，减少一些BN
调整数据补齐：将frame的前20条数据进行填充
5个随机种子 5折：0.788063492063492 模型较稳定
★根据类别频率统计class weight，加上线上效果0.788666
使用jitter进行数据增强：线下分数较高，线上0.779
加入LSTM-FCN之后，0.781
★数据增强 data enhance:对每折数据争抢很重要，否则全局会严重过拟合线上0.794

Resnet

resnet_acc0.798683635276431_combo0.82761327304097.csv
resnet_proba_t_0.798683635276431.csv
resnet_proba_x_0.798683635276431.csv

线上0.7457777777777777

MCNN

Multi-Scale Convolutional Neural Networks for Time Series Classification

针对现有时间序列分类方法的特征提取与分类过程分离，且无法提取存在于不同时间尺度序列的不同特征的问题，作者提出MCNN模型。
对于单一时间序列输入，进行降采样和滑动平均等变化，产生多组长度不同的时间序列，并在多组时间序列上进行卷积，提取不同时间尺度序列的特征。

失败的尝试

添加特征
lstm：也不算失败，目前网络结构比较简单，个人感觉应该LSTM的效果比CNN要好
超参数调优
加上归一化之后效果比较差

print('Scaler....')
for col in ['acc_x','acc_y','acc_z','acc_xg','acc_yg','acc_zg','mod','modg']:
    scaler = MinMaxScaler().fit(data[[col]])
    train[[col]] = scaler.transform(train[[col]])

加入特征

角度特征

# 角度
# data['acc_x_cos(x)'] = data['acc_x'] / data['mod']
# data['acc_y_cos(y)'] = data['acc_y'] / data['mod']
# data['acc_z_cos(z)'] = data['acc_z'] / data['mod']
# data['acc_x_angle(x)'] = np.arccos(data['acc_x_cos(x)'])
# data['acc_y_angle(y)'] = np.arccos(data['acc_y_cos(y)'])
# data['acc_z_angle(z)'] = np.arccos(data['acc_z_cos(z)'])

# 2020.7.8
data['mod2'] = data.acc_x ** 2 + data.acc_y ** 2 + data.acc_z ** 2
data['modg2'] = data.acc_xg ** 2 + data.acc_yg ** 2 + data.acc_zg ** 2

92/92 - 2s - loss: 0.0624 - acc: 0.9803 - val_loss: 0.8870 - val_acc: 0.7840 - lr: 3.1250e-05
Epoch 00117: early stopping
accuracy_score 0.7839506172839507 acc_combo 0.8136063753347688

线上0.7614603174603174

数据增强

datagen = tf.keras.preprocessing.image.ImageDataGenerator(
    height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
    horizontal_flip=True,  # randomly flip images
    vertical_flip=True  # randomly flip imag
)

线下0.792 线上0.7577619047619049

LSTM+CNN

92/92 - 2s - loss: 0.0656 - acc: 0.9798 - val_loss: 1.0423 - val_acc: 0.7599 - lr: 7.8125e-06
Epoch 138/500
92/92 - 2s - loss: 0.0594 - acc: 0.9832 - val_loss: 1.0431 - val_acc: 0.7613 - lr: 7.8125e-06
Epoch 00138: early stopping
accuracy_score 0.766803840877915 acc_combo 0.800509504213206

线上0.758

X = GlobalMaxPooling2D()(X)

Epoch 119/500
23/23 - 1s - loss: 0.1941 - acc: 0.9311 - val_loss: 0.9021 - val_acc: 0.7359 - lr: 6.2500e-05
Epoch 00119: early stopping
accuracy_score 0.7414266117969822 acc_combo 0.7816317199033243
5kflod mean acc score:0.728880483560249
5kflod mean combo score:0.7694991110919478

线上 0.7402857142857142

VGG19 预训练模型

accuracy_score 0.10150891632373114 acc_combo 0.1966490299823643
5kflod mean acc score:0.35815603637043997
5kflod mean combo score:0.4347271162644451

加入word2vec词向量特征

feainput = Input(shape=(w2v_fea))
    dense = Dense(32, activation='relu')(feainput)
    dense = BatchNormalization()(dense)
    dense = Dropout(0.2)(dense)
    dense = Dense(64, activation='relu')(dense)
    dense = Dropout(0.2)(dense)
    dense = Dense(128, activation='relu')(dense)
    dense = Dropout(0.2)(dense)
    dense = Dense(256, activation='relu')(dense)
    dense = BatchNormalization()(dense)

Disout 和Dropout类似，线下效果相比dropout差，线上0.787，另外收敛时间比较慢
EMN Time series classification with Echo Memory Networks 线下：0.444+ 可能因为和原论文实现差别太大的原因。。
MultiVariateCNN 每个变量单做一个输入
SRCNN

论文地址： Image Super-Resolution Using DeepConvolutional Networks

Name		Name	Last commit message	Last commit date
Latest commit History 109 Commits
data		data
dockers		dockers
images		images
paper		paper
result		result
trys		trys
.gitignore		.gitignore
README.md		README.md
cnn1d.py		cnn1d.py
cnn2d.py		cnn2d.py
ensemble.py		ensemble.py
extend.py		extend.py
har.py		har.py
har_en.py		har_en.py
load_data.py		load_data.py
load_inv_data.py		load_inv_data.py
utils.py		utils.py

China-ChallengeHub/Cellphone-Behavior

Folders and files

Latest commit

History

Repository files navigation

模型

CNN

LSTM

Multi Input

HAR：CNN输入与Dense输入融合

Resnet

失败的尝试

参考资料

About

Resources

Stars

Watchers

Forks

Languages