Permission denied error writing scan results

[howff]

After running for 9 hours WhiteRabbit crashed with a "Permission denied" error after printing "Generating scan report"

And it doesn't say which filename it was trying to write that failed

./whiteRabbit -ini ini
10:40:06 Started new scan
10:40:06 Scanning table one
Stopped after 1000 rows
etc.
(Some tables take 3 hours despite only loading 1000 rows !!!)
21:14:10 Generating scan report
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Permission denied
at org.apache.poi.xssf.streaming.SXSSFWorkbook.createAndRegisterSXSSFSheet(SXSSFWorkbook.java:662)
at org.apache.poi.xssf.streaming.SXSSFWorkbook.createSheet(SXSSFWorkbook.java:679)
at org.ohdsi.whiteRabbit.scan.SourceDataScan.createFieldOverviewSheet(SourceDataScan.java:201)
at org.ohdsi.whiteRabbit.scan.SourceDataScan.generateReport(SourceDataScan.java:182)
at org.ohdsi.whiteRabbit.scan.SourceDataScan.process(SourceDataScan.java:117)
at org.ohdsi.whiteRabbit.WhiteRabbitMain.launchCommandLine(WhiteRabbitMain.java:268)
at org.ohdsi.whiteRabbit.WhiteRabbitMain.(WhiteRabbitMain.java:126)
at org.ohdsi.whiteRabbit.WhiteRabbitMain.main(WhiteRabbitMain.java:121)
Caused by: java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at org.apache.poi.util.DefaultTempFileCreationStrategy.createTempFile(DefaultTempFileCreationStrategy.java:110)
at org.apache.poi.util.TempFile.createTempFile(TempFile.java:66)
at org.apache.poi.xssf.streaming.SheetDataWriter.createTempFile(SheetDataWriter.java:87)
at org.apache.poi.xssf.streaming.SheetDataWriter.(SheetDataWriter.java:70)
at org.apache.poi.xssf.streaming.SheetDataWriter.(SheetDataWriter.java:75)
at org.apache.poi.xssf.streaming.SXSSFWorkbook.createSheetDataWriter(SXSSFWorkbook.java:330)
at org.apache.poi.xssf.streaming.SXSSFSheet.(SXSSFSheet.java:80)
at org.apache.poi.xssf.streaming.SXSSFWorkbook.createAndRegisterSXSSFSheet(SXSSFWorkbook.java:658)
... 7 more

My ini file is like this:
WORKING_FOLDER = /home/myusername/WhiteRabbit_v0.10.3
DATA_TYPE = PostgreSQL
SERVER_LOCATION = 127.0.0.1/mydbname
USER_NAME = myusername
PASSWORD = mypassword
DATABASE_NAME = mydbname
DELIMITER = ,
TABLES_TO_SCAN = *
SCAN_FIELD_VALUES = yes
MIN_CELL_COUNT = 5
MAX_DISTINCT_VALUES = 1000
ROWS_PER_TABLE = 1000
CALCULATE_NUMERIC_STATS = yes
NUMERIC_STATS_SAMPLER_SIZE = 500

[MaximMoinat]

Hi @howff, apparently WR is not allowed to write to the given working folder. Could you try running WR as a super user or change the permissions to the folder?

To speed up the scan, you can disable SCAN_FIELD_VALUES values and/or disable CALCULATE_NUMERIC_STATS by setting them to 'no'.

[howff]

There's nothing wrong with the permissions on the WORKING_FOLDER.

I had to use strace to find the problem - WhiteRabbit tries to create /tmp/poifiles/poi-sxssf-sheetblah.xml but the /tmp/poifiles directory has insufficient permissions when multiple users run it.

[MaximMoinat]

Thanks for looking into the issue further. I was not aware that WR tries to write to a tmp folder. For now, I don't have a solution, but will take a look whether this behaviour can be changed in a future release.

[blootsvoets]

@MaximMoinat you could use this solution to force Apache POI to use a specific tmpdir:
https://stackoverflow.com/a/35453124/574082

File dir = new File("somepath");
dir.mkdir();
org.apache.poi.util.TempFile.setTempFileCreationStrategy(new DefaultTempFileCreationStrategy(dir));

[MaximMoinat]

Thanks, I will check it out and create a fix if needed.

…

On Tue, 23 Mar 2021 at 13:54, Joris Borgdorff ***@***.***> wrote: @MaximMoinat <https://github.com/MaximMoinat> you could use this solution to force Apache POI to use a specific tmpdir: https://stackoverflow.com/a/35453124/574082 File dir = new File("somepath"); dir.mkdir();org.apache.poi.util.TempFile.setTempFileCreationStrategy(new DefaultTempFileCreationStrategy(dir)); — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#293 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEH767EG5JETJYGGGBZQ5UTTFCFPJANCNFSM4ZKGYYLQ> .

[tom-dyar]

I do think it would be great that as soon as ANY file is known to be needed, at that point check to make sure the path is valid, and alert the user before a significant amount of time elapses -- I just ran into this with a misconfigured path to the output in my .ini file!

[howff]

Hi
This is still a blocker preventing multiple users from using WhiteRabbit!

[janblom]

I have uploaded a new version for testing: v0.10.9-test. Please see the explanation there.

It would be highly appreciated if someone can test this, and report success, or details of failure.