neural network training

[dadrake3]

Is there a way to terminate the training of a network if the error rate has converged to a certain significant figure?

[mc10011]

This sounds similar to the problem described in this link:

https://www.researchgate.net/post/Why_is_Pyrren_in_python_taking_forever_to_run

I have encountered this problem as well and added a "minimum error step criterium" in Train_LM function to alleviate it. The added criteria tackles the symptom, i.e. training running forever, and your request. My solution gives room for five more iterations until it breaks the training once and for all.

See my modified train_LM below. You can set the early stopping criteria as 1.0 instead of 5.0 and use the min_E_step as a convergence criteria for early stopping the training.

`def train_LM(P,Y,net,k_max=100,E_stop=1e-10,dampfac=3.0,dampconst=10.0,
verbose = False,min_E_step=1e-09):
""" Implementation of the Levenberg-Marquardt-Algorithm (LM) based on:
Levenberg, K.: A Method for the Solution of Certain Problems in Least Squares.
Quarterly of Applied Mathematics, 2:164-168, 1944.
and
Marquardt, D.: An Algorithm for Least-Squares Estimation of Nonlinear Parameters.
SIAM Journal, 11:431-441, 1963.

Args:

	P:		NN Inputs
	Y:		NN Targets
	net: 	neural network
	k_max:	maxiumum number of iterations
	E_stop:	Termination Error, Training stops when the Error <= E_stop
	dampconst:	constant to adapt damping factor of LM
	dampfac:	damping factor of LM
	min_E_step: minimum step for error. When reached 5 times, training terminates.        
Returns:
	net: 	trained Neural Network 
"""
#create data dict
data,net = prepare_data(P,Y,net)

#Calculate Jacobian, Error and error vector for first iteration
J,E,e = RTRL(net,data)
k = 0
ErrorHistory=np.zeros(k_max+1) #Vektor for Error hostory
ErrorHistory[k]=E
if verbose:
	print('Iteration: ',k,'		Error: ',E,'	scale factor: ',dampfac)

early=0

while True:
#run loop until either k_max or E_stop is reached

	JJ = np.dot(J.transpose(),J) #J.transp * J
	w = net['w'] #weight vector
	while True:
	#repeat until optimizing step is successful
		#gradient
		g = np.dot(J.transpose(),e)
		
		#calculate scaled inverse hessian
		try:
			G = np.linalg.inv(JJ+dampfac*np.eye(net['N'])) #scaled inverse hessian
		except np.linalg.LinAlgError:
			# Not invertible. Go small step in gradient direction
			w_delta = 1.0/1e10 * g
		else:
			# calculate weight modification
			w_delta = np.dot(-G,g)
		
		net['w'] = w + w_delta #new weight vector
		
		Enew = calc_error(net,data) #calculate new Error E			
		if Enew<E and abs(E-Enew)>=min_E_step:
		#Optimization Step successful!
			#if E-Enew<=1e-09:
			dampfac= dampfac/dampconst#adapt scale factor
			break #go to next iteration
		else:
		#Optimization Step NOT successful!\
			dampfac = dampfac*dampconst#adapt scale factor
			if abs(E-Enew)<=min_E_step:
				if verbose:
					print('E-Enew<=min_E_step Encountered!!')
				early=early+1
				if early>=5.0:
					if verbose:
						print('5 Times * E-Enew<=min_E_step Encountered!!')
				break                    
	
	#Calculate Jacobian, Error and error vector for next iteration
	J,E,e = RTRL(net,data)
	k = k+1
	ErrorHistory[k] = E
	if verbose:
		print('Iteration: ',k,'		Error: ',E,'	scale factor: ',dampfac)

	#Ceck if termination condition is fulfilled
	if k>=k_max:
		print('Maximum number of iterations reached')
		break
	elif E<=E_stop:
		print('Termination Error reached')
		break
	elif early>=5.0:
		print('Error decreased 5 times by minimum step. Force training exit.')
		break
    
net['ErrorHistory'] = ErrorHistory[:k]
return net`