Skip to content

TODO: forloop read #1010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Baunsgaard wants to merge 1 commit into from
Closed

TODO: forloop read #1010

Baunsgaard wants to merge 1 commit into from

Conversation

@Baunsgaard
Copy link
Contributor

@Baunsgaard Baunsgaard commented Aug 7, 2020 •
edited
Loading

This PR showcase a current limitation of the system where if one construct the sting path to read the system is unable to read the given file. It is not an issue if the path is created outside a for loop but in connection with the for loop the reading fails.

The PR does not contain a fix for the given problem.

@Baunsgaard
Copy link
Contributor Author

Baunsgaard commented Aug 7, 2020

Produce error:

20/08/07 15:34:48 WARN parser.Expression: WARNING: ./src/test/scripts/functions/io/forLoopRead.dml [line 24:4] -> v = read("src/test/scripts/functions/io/in/" +x + ".csv", rows=1, cols=1, format="csv" ) -- Parameter iofilename only supports a const string or const string concatenations.
20/08/07 15:34:48 WARN parser.Expression: WARNING: ./src/test/scripts/functions/io/forLoopRead.dml [line 24:4] -> v = read("src/test/scripts/functions/io/in/" +x + ".csv", rows=1, cols=1, format="csv" ) -- Read input file does not exist on FS (local mode): src/test/scripts/functions/io/in/.csv
20/08/07 15:34:48 WARN parser.Expression: WARNING: ./src/test/scripts/functions/io/forLoopRead.dml [line 24:4] -> v = read("src/test/scripts/functions/io/in/" +x + ".csv", rows=1, cols=1, format="csv" ) -- Read input file does not exist on FS (local mode): src/test/scripts/functions/io/in/.csv

java.lang.AssertionError:
failed to run script: ./src/test/scripts/functions/io/forLoopRead.dml
Standard Out:
SystemDS Statistics:
Total execution time: 0.022 sec.
Number of executed Spark inst: 0.

StackTrace:
LEVEL : 0
Exception : class org.apache.sysds.runtime.DMLRuntimeException
Message : org.apache.sysds.runtime.DMLRuntimeException: ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block
0 > org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:155)
0 > org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:87)
0 > org.apache.sysds.api.DMLScript.execute(DMLScript.java:395)
0 > org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:244)
0 > org.apache.sysds.api.DMLScript.main(DMLScript.java:160)
0 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1194)
0 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1103)
0 > org.apache.sysds.test.functions.io.ForLoopRead.testSimple(ForLoopRead.java:49)
0 > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
0 > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
0 > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
0 > java.lang.reflect.Method.invoke(Method.java:498)

... Stopping Stack Trace at JUnit
LEVEL : 1
Exception : class org.apache.sysds.runtime.DMLRuntimeException
Message : ERROR: Runtime error in for program block generated from for statement block between lines 0 and 0 -- Error evaluating for program block
1 > org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:156)
1 > org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:149)
1 > org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:87)
1 > org.apache.sysds.api.DMLScript.execute(DMLScript.java:395)
1 > org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:244)
1 > org.apache.sysds.api.DMLScript.main(DMLScript.java:160)
1 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1194)
1 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1103)
1 > org.apache.sysds.test.functions.io.ForLoopRead.testSimple(ForLoopRead.java:49)
1 > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
1 > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
1 > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
1 > java.lang.reflect.Method.invoke(Method.java:498)
... Stopping Stack Trace at JUnit
LEVEL : 2
Exception : class org.apache.sysds.runtime.DMLRuntimeException
Message : ERROR: Runtime error in program block generated from statement block between lines 24 and 25 -- Error evaluating instruction: SPARK°csvrblk°pREADv·MATRIX·FP64°_mVar6·MATRIX·FP64°1000°false°,°true°0.0°
2 > org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:279)
2 > org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204)
2 > org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
2 > org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:137)
2 > org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:149)
2 > org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:87)
2 > org.apache.sysds.api.DMLScript.execute(DMLScript.java:395)
2 > org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:244)
2 > org.apache.sysds.api.DMLScript.main(DMLScript.java:160)
2 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1194)
2 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1103)
2 > org.apache.sysds.test.functions.io.ForLoopRead.testSimple(ForLoopRead.java:49)
2 > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2 > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2 > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2 > java.lang.reflect.Method.invoke(Method.java:498)
... Stopping Stack Trace at JUnit
LEVEL : 3
Exception : class org.apache.sysds.runtime.DMLRuntimeException
Message : Reading of src/test/scripts/functions/io/in/.csv (102052782) failed.
3 > org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireReadIntern(CacheableData.java:515)
3 > org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:446)
3 > org.apache.sysds.runtime.controlprogram.caching.MatrixObject.acquireRead(MatrixObject.java:403)
3 > org.apache.sysds.hops.recompile.Recompiler.executeInMemoryMatrixReblock(Recompiler.java:1577)
3 > org.apache.sysds.runtime.instructions.spark.CSVReblockSPInstruction.processInstruction(CSVReblockSPInstruction.java:118)
3 > org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:247)
3 > org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204)
3 > org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
3 > org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:137)
3 > org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:149)
3 > org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:87)
3 > org.apache.sysds.api.DMLScript.execute(DMLScript.java:395)
3 > org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:244)
3 > org.apache.sysds.api.DMLScript.main(DMLScript.java:160)
3 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1194)
3 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1103)
3 > org.apache.sysds.test.functions.io.ForLoopRead.testSimple(ForLoopRead.java:49)
3 > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
3 > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
3 > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
3 > java.lang.reflect.Method.invoke(Method.java:498)
... Stopping Stack Trace at JUnit
LEVEL : 4
Exception : class org.apache.hadoop.mapred.InvalidInputException
Message : Input path does not exist: file:/home/baunsgaard/github/systemds/src/test/scripts/functions/io/in/.csv
4 > org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
4 > org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
4 > org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
4 > org.apache.sysds.runtime.io.ReaderTextCSVParallel.readMatrixFromHDFS(ReaderTextCSVParallel.java:88)
4 > org.apache.sysds.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:234)
4 > org.apache.sysds.runtime.util.DataConverter.readMatrixFromHDFS(DataConverter.java:187)
4 > org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:470)
4 > org.apache.sysds.runtime.controlprogram.caching.MatrixObject.readBlobFromHDFS(MatrixObject.java:65)
4 > org.apache.sysds.runtime.controlprogram.caching.CacheableData.readBlobFromHDFS(CacheableData.java:950)
4 > org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireReadIntern(CacheableData.java:497)
4 > org.apache.sysds.runtime.controlprogram.caching.CacheableData.acquireRead(CacheableData.java:446)
4 > org.apache.sysds.runtime.controlprogram.caching.MatrixObject.acquireRead(MatrixObject.java:403)
4 > org.apache.sysds.hops.recompile.Recompiler.executeInMemoryMatrixReblock(Recompiler.java:1577)
4 > org.apache.sysds.runtime.instructions.spark.CSVReblockSPInstruction.processInstruction(CSVReblockSPInstruction.java:118)
4 > org.apache.sysds.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:247)
4 > org.apache.sysds.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:204)
4 > org.apache.sysds.runtime.controlprogram.BasicProgramBlock.execute(BasicProgramBlock.java:125)
4 > org.apache.sysds.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:137)
4 > org.apache.sysds.runtime.controlprogram.Program.execute(Program.java:149)
4 > org.apache.sysds.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:87)
4 > org.apache.sysds.api.DMLScript.execute(DMLScript.java:395)
4 > org.apache.sysds.api.DMLScript.executeScript(DMLScript.java:244)
4 > org.apache.sysds.api.DMLScript.main(DMLScript.java:160)
4 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1194)
4 > org.apache.sysds.test.AutomatedTestBase.runTest(AutomatedTestBase.java:1103)
4 > org.apache.sysds.test.functions.io.ForLoopRead.testSimple(ForLoopRead.java:49)
4 > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
4 > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
4 > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
4 > java.lang.reflect.Method.invoke(Method.java:498)
... Stopping Stack Trace at JUnit

@Baunsgaard Baunsgaard closed this Mar 7, 2021
@Baunsgaard Baunsgaard deleted the ForLoopRead branch March 7, 2021 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Baunsgaard