Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logistic Regression Errors #103

Closed
sysuxiaofan opened this issue Jul 3, 2018 · 30 comments
Closed

Logistic Regression Errors #103

sysuxiaofan opened this issue Jul 3, 2018 · 30 comments

Comments

@sysuxiaofan
Copy link

Caught ExecuteError in logistic regression. Details:
ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).
Traceback
File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute
gp.Combine_sa(Input_Combine_rasters, thmUC)

Error Info:
ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).

@trotor
Copy link
Collaborator

trotor commented Jul 3, 2018 via email

@sysuxiaofan
Copy link
Author

sysuxiaofan commented Jul 3, 2018

Executing: LogisticRegressionTool bwof_gch1;bwof_ghy1;bwof_int1;bwof_sta1;bwof_str1 f;f;f;f;f F:\data_processed\ltemp\bwof_gch1_Weights.dbf;F:\data_processed\ltemp\bwof_ghy1_Weights.dbf;F:\data_processed\ltemp\bwof_int1_Weights.dbf;F:\data_processed\ltemp\bwof_sta1_Weights.dbf;F:\data_processed\ltemp\bwof_str1_Weights.dbf cumo -99 4 F:\data_processed\temp\blr_ply.dbf F:\data_processed\temp\blr_coef.dbf F:\data_processed\temp\blr_post.tif F:\data_processed\temp\blr_poststd.tif F:\data_processed\temp\blr_conf.tif
Start Time: Tue Jul 03 15:14:23 2018
Running script LogisticRegressionTool...
Checking environment...
Input rasters: [u'bwof_gch1', u'bwof_ghy1', u'bwof_int1', u'bwof_sta1', u'bwof_str1']
Evidence_types: [u'f', u'f', u'f', u'f', u'f']
Wts_Tables: [u'F:\data_processed\ltemp\bwof_gch1_Weights.dbf', u'F:\data_processed\ltemp\bwof_ghy1_Weights.dbf', u'F:\data_processed\ltemp\bwof_int1_Weights.dbf', u'F:\data_processed\ltemp\bwof_sta1_Weights.dbf', u'F:\data_processed\ltemp\bwof_str1_Weights.dbf']
F:\data_processed\ltemp\bwof_gch1_Weights.dbf
F:\data_processed\ltemp\bwof_ghy1_Weights.dbf
F:\data_processed\ltemp\bwof_int1_Weights.dbf
F:\data_processed\ltemp\bwof_sta1_Weights.dbf
F:\data_processed\ltemp\bwof_str1_Weights.dbf
TrainPts: cumo
MissingDataValue: -99
thmUC: F:\data_processed\ltemp\tmp_UCras0

========== arcsdm values ==========
ArcSdm 5.01.05 for ArcGis (Pro and Desktop)

Arcgis environment: Desktop (10.2)
Workspace: F:\data_processed\ltemp (FileSystem)
Scratch workspace: F:\data_processed\ltemp (FileSystem)
Map Units: meter
Mask: "bwof_gch1" and it is RasterLayer
Mask size: 52296.0
Cell Size: 2000

Training Sites: 12

Unit Cell Area: 4.0km^2, Cells in area: 13074.0
Prior Probability: 0.000918
Training Set: F:\data_processed\2016-10-13\weights\cumo.shp
Study Area Raster: F:\data_processed\2016-10-13\weights\bwof_gch1
Study Area Area: 52296.0km^2
Creating Generalized Class rasters.
Output_Raster: F:\data_processed\ltemp\bwof_gch1_G44
Temp_Raster: F:\data_processed\ltemp\tmp_rst220
Output_Raster: F:\data_processed\ltemp\bwof_ghy1_G44
Temp_Raster: F:\data_processed\ltemp\tmp_rst221
Output_Raster: F:\data_processed\ltemp\bwof_int1_G44
Temp_Raster: F:\data_processed\ltemp\tmp_rst222
Output_Raster: F:\data_processed\ltemp\bwof_sta1_G44
Temp_Raster: F:\data_processed\ltemp\tmp_rst223
Output_Raster: F:\data_processed\ltemp\bwof_str1_G44
Temp_Raster: F:\data_processed\ltemp\tmp_rst224
Combining...F:\data_processed\ltemp\bwof_gch1_G44;F:\data_processed\ltemp\bwof_ghy1_G44;F:\data_processed\ltemp\bwof_int1_G44;F:\data_processed\ltemp\bwof_sta1_G44;F:\data_processed\ltemp\bwof_str1_G44

Caught ExecuteError in logistic regression. Details:
ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).

Traceback
File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute
gp.Combine_sa(Input_Combine_rasters, str(thmUC))
Error Info:
ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).

Traceback (most recent call last):
File "", line 1562, in execute
File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute
gp.Combine_sa(Input_Combine_rasters, str(thmUC))
ExecuteError: ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).

Failed to execute (LogisticRegressionTool).
Failed at Tue Jul 03 15:16:03 2018 (Elapsed Time: 1 minutes 40 seconds)

@sysuxiaofan
Copy link
Author

ArcGIS 10.2, five WofE layers.

@trotor
Copy link
Collaborator

trotor commented Jul 4, 2018

Ok, please create empty text file called DEBUG into toolbox\arcsdm\ directory and rerun the project for more detailed output. It could be some error in parameters or perhaps incompatibility with 10.2.

@sysuxiaofan
Copy link
Author

I modified the following codes:
#thmUC = gp.createscratchname("tmp_UCras", '', 'raster', gp.scratchworkspace) # Xiao Fan modified
thmUC ="r""+ gp.createscratchname("tmp_UCras", '', 'raster', gp.scratchworkspace) + """ # Xiao Fan modified
#Input_Combine_rasters =";".join(Wts_Rasters) # Xiao Fan modified
Input_Combine_rasters ="""+"r'" + "'; r'".join(Wts_Rasters) + "'"+""" # Xiao Fan modified
The original bugs seem have been repaired. But, it shows the following erros
Caught ExecuteError in logistic regression. Details:
Failed to execute. Parameters are not valid.
ERROR 000878: Output raster: The length of the grid base name in F:\data_processed\ltemp\r"F:\data_processed\ltemp\tmp_UCras0" is longer than 13.
Failed to execute (Combine).

Traceback
File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 164, in Execute
gp.Combine_sa(Input_Combine_rasters, thmUC) # Xiao Fan modified

Error Info:
Failed to execute. Parameters are not valid.
ERROR 000878: Output raster: The length of the grid base name in F:\data_processed\ltemp\r"F:\data_processed\ltemp\tmp_UCras0" is longer than 13.
Failed to execute (Combine).

Traceback (most recent call last):
File "", line 1562, in execute
File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 164, in Execute
gp.Combine_sa(Input_Combine_rasters, thmUC) # Xiao Fan modified
ExecuteError: Failed to execute. Parameters are not valid.
ERROR 000878: Output raster: The length of the grid base name in F:\data_processed\ltemp\r"F:\data_processed\ltemp\tmp_UCras0" is longer than 13.
Failed to execute (Combine).

Failed to execute (LogisticRegressionTool).

@sysuxiaofan
Copy link
Author

And, I have tried to short the file name (tmp_UCras0, e.g. UCr), but the erros still exists.

@sysuxiaofan
Copy link
Author

I have referenced the ArcGIS help, but I don't think my new codes have any problem.

Combine_sample.py
Description:
Combines multiple rasters such that a unique output value is
assigned to each unique combination of input values.
Requirements: None
Author: ESRI
Date: Sept 6, 2005
Import system modules
import arcgisscripting
Create the Geoprocessor object
gp = arcgisscripting.create()
try:

Set local variables

outRaster = "C:/data/final_1"

Check out Spatial Analyst extension license

gp.CheckOutExtension("Spatial")

Process: Combine...

gp.Combine_sa("'C:/data/ras_1';'C:/data/ras_2'", outRaster)
except:

If an error occurred while running a tool, then print the messages.

print gp.GetMessages()

@trotor
Copy link
Collaborator

trotor commented Jul 4, 2018

What is the output of this (print it after it is set):
Input_Combine_rasters ="""+"r'" + "'; r'".join(Wts_Rasters) + "'"+""" # Xiao Fan modified

@sysuxiaofan
Copy link
Author

Input_Combine_rasters ="r'F:\data_processed\ltemp\bwof_gch1_G1'; r'F:\data_processed\ltemp\bwof_ghy1_G1'; r'F:\data_processed\ltemp\bwof_int1_G1'; r'F:\data_processed\ltemp\bwof_sta1_G1'; r'F:\data_processed\ltemp\bwof_str1_G1'"

gp.Combine_sa("'C:/data/ras_1';'C:/data/ras_2'", outRaster)

@sysuxiaofan
Copy link
Author

using r to make \t or \r not work.

@sysuxiaofan
Copy link
Author

and thmUC=r"F:\data_processed\ltemp\tmp_UCras0"

@trotor
Copy link
Collaborator

trotor commented Jul 4, 2018 via email

@sysuxiaofan
Copy link
Author

Caught ExecuteError in logistic regression. Details:
ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).

Traceback
File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 165, in Execute
gp.Combine_sa("'F:/data_processed/ltemp/bwof_gch1_G0';'F:/data_processed/ltemp/bwof_ghy1_G0';'F:/data_processed/ltemp/bwof_int1_G0';'F:/data_processed/ltemp/bwof_sta1_G0';'F:/data_processed/ltemp/bwof_str1_G0'", "F:/data_processed/ltemp/tmp_UCras0") # Xiao Fan modified

Error Info:
ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).

Traceback (most recent call last):
File "", line 1562, in execute
File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 165, in Execute
gp.Combine_sa("'F:/data_processed/ltemp/bwof_gch1_G0';'F:/data_processed/ltemp/bwof_ghy1_G0';'F:/data_processed/ltemp/bwof_int1_G0';'F:/data_processed/ltemp/bwof_sta1_G0';'F:/data_processed/ltemp/bwof_str1_G0'", "F:/data_processed/ltemp/tmp_UCras0") # Xiao Fan modified
ExecuteError: ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).

Failed to execute (LogisticRegressionTool).
Failed at Wed Jul 04 22:22:06 2018 (Elapsed Time: 13.32 seconds)

@trotor
Copy link
Collaborator

trotor commented Jul 4, 2018 via email

@Eliasmgprado
Copy link

Eliasmgprado commented Jul 18, 2018

I had a hard time running the logistiregression.py script.

The main problems were:

  • Arcmap crashes when running AddJoin_management function
  • Files are not correctly identified
  • Some unecessary lines giving error
  • con_sa () function giving error

some of these problems have already been reported in this post.
after a lot of work, I was able to solve the problems and finally run the script successfully.

Here are the script modifications i need to do:

  • added on line 25 the import of spatial analyst functions
from arcpy.sa import *
  • modified the lines 94 and 95:
Input_Rasters[i] = s.strip("'");
if arcsdm.common.testandwarn_filegeodatabase_source(s.strip("'")):
  • modified lines 141 to 148 :
Temp_Raster = gp.CreateScratchName('tmp_rst', '', 'rst',  gp.scratchworkspace)
gp.copyraster_management(Input_Raster, Temp_Raster)
gp.JoinField_management(Temp_Raster, 'Value', Wts_Table, 'CLASS')
gp.AddMessage('Temp_Raster: %s'%(str(Temp_Raster)))
gp.Lookup_sa(Temp_Raster, "GEN_CLASS", Output_Raster)
  • commented the lines 461, 462, 464, 466
strLine = fLR.readline()
#vTabUCrows = workarounds_93.rowgen(gp.SearchCursor(vTabUC))
#vTabUCrow = vTabUCrows.Next()
#ttl = 0
#while vTabUCrow:
#for vTabUCrow in vTabUCrows: ttl += 1
  • comment line 582:
#gp.makerasterlayer_management(cmb, cmbrl)
  • modified lines 587 to 589:
cmb_cpy = gp.createscratchname("cmb_cpy", '', 'raster', arcpy.env.scratchFolder)
gp.copyraster_management(cmb, cmb_cpy)
gp.JoinField_management(cmb_cpy, 'Value', tbltv, 'ID')
  • modified lines 607 to 609:
outcon1 = Con(cmb_cpy, Lookup(cmb_cpy,"LRPOSTPROB"),"0","LRPOSTPROB > 0")
outcon1.save(outRaster1)
outcon2 = Con(cmb_cpy,Lookup(cmb_cpy,"LR_STD_DEV"),"0","LR_STD_DEV > 0")
outcon2.save(outRaster2)
outcon3 = Con(cmb_cpy,Lookup(cmb_cpy,"LRTVALUE"),"0","LRTVALUE > 0")
outcon3.save(outRaster3)

Running on ArcMap 10.5.1

@trotor
Copy link
Collaborator

trotor commented Jul 19, 2018 via email

@trotor
Copy link
Collaborator

trotor commented Jul 20, 2018

Could you send me the fixed .py file here as attachement? Others could benefit from your valuable work!

@trotor trotor added the bug label Jul 20, 2018
@Eliasmgprado
Copy link

Here is the file, code needs cleaning of some unecessary commented lines.
logisticregression.zip

@trotor
Copy link
Collaborator

trotor commented Sep 7, 2018

Update on this issue: I think I haven't yet merged the code - we had summer holidays and some computer related problems. Will need to check and do when smoke clears.

@trotor
Copy link
Collaborator

trotor commented Sep 25, 2018

Hi, merged these changes - if you have time, please test if it went correctly.

@Mahfoudb
Copy link

Caught ExecuteError in logistic regression. Details:
ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).
Traceback
File "G:\360Downloads\ArcSDM\Toolbox\arcsdm\logisticregression.py", line 162, in Execute
gp.Combine_sa(Input_Combine_rasters, thmUC)

Error Info:
ERROR 010328: Syntax error at or near symbol ,.
ERROR 010267: Syntax error in parsing grid expression.
Failed to execute (Combine).

Hello,
Have you found a solution for this problem because I get the same one.

@Mahfoudb
Copy link

Can you help me please to solve this problem

@trotor
Copy link
Collaborator

trotor commented Nov 14, 2019

Can you help me please to solve this problem

What inputs are you using?

@Mahfoudb
Copy link

Executing: LogisticRegression pnt;rh;ocp;lth;frc;exp;elv f;f;f;f;f;f;f pnt_ct;rh_ct;ocp_ct;lth_ct;frc_ct;exp_ct;elv_ct TP -99 1 C:\Data\LR_logpol.dbf C:\Data\LR_Coeff.dbf C:\Data\LR_pprb C:\Data\LR_std C:\Data\LR_conf
Start Time: Fri Nov 15 02:33:26 2019
Running script LogisticRegression...
Workspace: C:\Data
Scratch workspace: C:\temp
Prior Probability: 0.103672
Training Set: C:\Data\TP.shp
Number of Training Sites: 107
Study Area Raster: C:\Data\mask
Study Area Area (sq km): 1032.1034
Unit Cell Area (sq km): 1.0
Map Units: meter
GP ERRORS:
ERROR 010070: Raster read error.
ERROR 010067: Error in executing grid expression.
Failed to execute (Combine).

PYTHON ERRORS:
Traceback Info:
File "C:\SDM\Scripts\LogisticRegressionAll.py", line 97, in
gp.Combine_sa(Input_Combine_rasters, thmUC)

Error Info:
<class 'arcgisscripting.ExecuteError'>: ERROR 010070: Raster read error.
ERROR 010067: Error in executing grid expression.
Failed to execute (Combine).

Failed script LogisticRegression...

Traceback (most recent call last):
File "C:\SDM\Scripts\LogisticRegressionAll.py", line 753, in
raise 'Stopped on error.'
TypeError: exceptions must be old-style classes or derived from BaseException, not str

Failed to execute (LogisticRegression).
Failed at Fri Nov 15 02:34:09 2019 (Elapsed Time: 43.23 seconds)

@Mahfoudb
Copy link

This was the error message. I Have used Arsdm many times before with no problem. and when I bring the old data and run logistic regression with those old data it works.. I wonder if there is any restriction on the study area surface?

@Mahfoudb
Copy link

Executing: LogisticRegression pnt;rh;ocp;lth;frc;exp;elv c;c;c;c;c;c;c pnt_ct;rh_ct;ocp_ct;lth_ct;frc_ct;exp_ct;elv_ct TP -99 1 C:\Data\LR_logpol.dbf C:\Data\LR_Coeff.dbf C:\Data\LR_pprb C:\Data\LR_std C:\Data\LR_conf
Start Time: Fri Nov 15 02:43:47 2019
Running script LogisticRegression...
Workspace: C:\Data
Scratch workspace: C:\Tmp
Prior Probability: 0.103672
Training Set: C:\Data\TP.shp
Number of Training Sites: 107
Study Area Raster: C:\Data\mask
Study Area Area (sq km): 1032.1034
Unit Cell Area (sq km): 1.0
Map Units: meter
Running C:\Tmp\sdmlr.bat:
Logistic regression output file C:\Tmp\logpol.tba does not exist.
Error in case.dat or param.dat.
GP ERRORS:

PYTHON ERRORS:
Traceback Info:
File "C:\SDM\Scripts\LogisticRegressionAll.py", line 571, in
raise Exception

Error Info:
<type 'exceptions.Exception'>:

Failed script LogisticRegression...

Traceback (most recent call last):
File "C:\SDM\Scripts\LogisticRegressionAll.py", line 753, in
raise 'Stopped on error.'
TypeError: exceptions must be old-style classes or derived from BaseException, not str

Failed to execute (LogisticRegression).
Failed at Fri Nov 15 02:45:40 2019 (Elapsed Time: 1 minutes 53 seconds)

@Mahfoudb
Copy link

Sometimes it gives this 2nd error

@trotor
Copy link
Collaborator

trotor commented Nov 15, 2019

There is no restriction that I know of, but the tools seemed to be picky sometimes. Can you do operations fine on them manually? Unfortunately I have no clue on the cause. Anyone else?

@artolaiho
Copy link
Collaborator

Hello, try to set workspace and scratch workspace to same folder.
You can also test my fixes on arto-dev branch. See readme.

@artolaiho
Copy link
Collaborator

Load the newest version of ArcSDM (V5.02) and test again. If the problem persists, reopen this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants