hyperparameter in textRNN

[longbowking]

text_classification/a03_TextRNN/p8_TextRNN_model.py

Lines 74 to 83 in 68e2fcf

	def loss(self,l2_lambda=0.0001):
	with tf.name_scope("loss"):
	#input: `logits` and `labels` must have the same shape `[batch_size, num_classes]`
	#output: A 1-D `Tensor` of length `batch_size` of the same type as `logits` with the softmax cross entropy loss.
	losses = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=self.input_y, logits=self.logits);#sigmoid_cross_entropy_with_logits.#losses=tf.nn.softmax_cross_entropy_with_logits(labels=self.input_y,logits=self.logits)
	#print("1.sparse_softmax_cross_entropy_with_logits.losses:",losses) # shape=(?,)
	loss=tf.reduce_mean(losses)#print("2.loss.loss:", loss) #shape=()
	l2_losses = tf.add_n([tf.nn.l2_loss(v) for v in tf.trainable_variables() if 'bias' not in v.name]) * l2_lambda
	loss=loss+l2_losses
	return loss

• the final loss is defined as the sum of cross_entropy_loss and L2-loss to penalty large variables, where a hyperparameter l2_lambda is used to balance these two items. In the algorithm, as the hyperparameter l2_lambda is directly set to 0.0001, would it be too small(or too large) under some circumstances such that one in the two items loses its contribution to the whole loss?
• in general, is there any pratical methods to guide me to set values of hyperparameters, e.g. l2_lambda
  @brightmart

[brightmart]

hi, you can search l2_lambda among 0.01,0.001,0.0001, and choose the value that give you lowerest validation loss. bright

…

________________________________________ 发件人: longbowking [notifications@github.com] 发送时间: 2018年1月27日 23:09 收件人: brightmart/text_classification 抄送: brightmart; Mention 主题: [brightmart/text_classification] on the hyperparameter in p8_text (#32) https://github.com/brightmart/text_classification/blob/68e2fcf57a8dcec7e7d12f78953ed570451f0076/a03_TextRNN/p8_TextRNN_model.py#L74-L83 * the final loss is defined as the sum of cross_entropy_loss and L2-loss to penalty large variables, where a hyperparameter l2_lambda is used to balance these two items. In the algorithm, as the hyperparameter l2_lambda is directly set to 0.0001, would it be too small(or too large) under some circumstances such that one in the two items loses its contribution to the whole loss? * in general, is there any pratical methods to guide me to set values of hyperparameters, e.g. l2_lambda @brightmart<https://github.com/brightmart> ― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#32>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASuYMFIAG35PYMjDHWpisLVYjg0ci8Pnks5tOzwlgaJpZM4RvUqf>.