Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] FileAlreadyExistsException in IndexedCSVDataSink #1043

Closed
ChrizZz110 opened this issue Oct 12, 2018 · 1 comment
Closed

[Bug] FileAlreadyExistsException in IndexedCSVDataSink #1043

ChrizZz110 opened this issue Oct 12, 2018 · 1 comment
Assignees
@p-f
Labels
bug output

Comments

@ChrizZz110
Copy link
Contributor

ChrizZz110 commented Oct 12, 2018 •
edited
Loading

While writing some bigger graph (e.g. > 1GB) to a IndexedCSVDataSink, a FileAlreadyExistsException occurs with the following trace:

Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: java.nio.file.FileAlreadyExistsException: File already exists: /home/christopher/datasets/stackoverflow_csv_sampled_indexed_collection/edges/commenttoquestion/data.csv
	at org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:625)
	at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:234)
	at org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
	at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:816)
	at org.gradoop.examples.AbstractRunner.writeGraphCollection(AbstractRunner.java:162)
	at org.gradoop.utils.converter.GradoopFormatConverter.main(GradoopFormatConverter.java:47)
Caused by: java.nio.file.FileAlreadyExistsException: File already exists: /home/christopher/datasets/stackoverflow_csv_sampled_indexed_collection/edges/commenttoquestion/data.csv
	at org.apache.flink.core.fs.local.LocalFileSystem.create(LocalFileSystem.java:266)
	at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.create(SafetyNetWrapperFileSystem.java:121)
	at org.gradoop.flink.io.impl.csv.indexed.functions.MultipleFileOutputFormat.getAndCreateFileStream(MultipleFileOutputFormat.java:316)
	at org.gradoop.flink.io.impl.csv.indexed.functions.IndexedCSVFileFormat.mapWriter(IndexedCSVFileFormat.java:154)
	at org.gradoop.flink.io.impl.csv.indexed.functions.IndexedCSVFileFormat.writeRecord(IndexedCSVFileFormat.java:214)
	at org.gradoop.flink.io.impl.csv.indexed.functions.IndexedCSVFileFormat.writeRecord(IndexedCSVFileFormat.java:44)
	at org.apache.flink.runtime.operators.DataSinkTask.invoke(DataSinkTask.java:218)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
	at java.lang.Thread.run(Thread.java:748)

It is looking like a race problem since two executors try to create the same file.
@p-f
Copy link
Collaborator

p-f commented Oct 18, 2018

I'll have a look (haven't found anything yet).

p-f added a commit to p-f/gradoop that referenced this issue Oct 19, 2018
@p-f
[dbs-leipzig#1043] Fix IndexedCSVFileFormat.
f94c84d
p-f added a commit to p-f/gradoop that referenced this issue Oct 19, 2018
@p-f
[dbs-leipzig#1043] Fix some typos.
66c0717
@p-f p-f mentioned this issue Oct 19, 2018
Closed
p-f added a commit to p-f/gradoop that referenced this issue Oct 22, 2018
@p-f
[dbs-leipzig#1043] Typos and remove unused constant.
c8852ce 
The constant is accessible via Flink API.
p-f added a commit to p-f/gradoop that referenced this issue Oct 22, 2018
@p-f
[dbs-leipzig#1043] Fix the indexed CSV sink.
24e3230
@p-f p-f mentioned this issue Oct 22, 2018
Closed
p-f added a commit to p-f/gradoop that referenced this issue Oct 26, 2018
@p-f
[dbs-leipzig#1043] Always create directories instead of files.
bcf46c3
p-f added a commit to p-f/gradoop that referenced this issue Nov 20, 2018
@p-f
[dbs-leipzig#1043] Get the parallelism of subtasks from the runtime c…
035fdc3 
…ontext.
timo95 added a commit to timo95/gradoop that referenced this issue Nov 22, 2018
@timo95
[dbs-leipzig#1043] Fix format creation and overwrite rules
0674a40
@timo95 timo95 mentioned this issue Nov 22, 2018
Merged
timo95 added a commit to timo95/gradoop that referenced this issue Nov 22, 2018
@timo95
[dbs-leipzig#1043] Fix style
f143565
timo95 added a commit to timo95/gradoop that referenced this issue Nov 22, 2018
@timo95
[dbs-leipzig#1043] Fix JavaDoc
9e7d3a1
timo95 added a commit to timo95/gradoop that referenced this issue Nov 26, 2018
@timo95
[dbs-leipzig#1043] Fix PR comments
3890041
timo95 added a commit to timo95/gradoop that referenced this issue Nov 26, 2018
@timo95
[dbs-leipzig#1043] Fix overwrite on local fs
c8f3cc3
timo95 added a commit to timo95/gradoop that referenced this issue Nov 26, 2018
@timo95
[dbs-leipzig#1043] Fix bug
1cf1548
ChrizZz110 pushed a commit that referenced this issue Nov 26, 2018
@timo95 @ChrizZz110
[#1043] Fix the indexed CSV sink. (#1094)
95f7c90 
fixes #1043
0x002A pushed a commit to ChrizZz110/gradoop that referenced this issue Feb 19, 2019
@timo95 @ChrizZz110
[dbs-leipzig#1043] Fix the indexed CSV sink. (dbs-leipzig#1094)
fa98d56 
fixes dbs-leipzig#1043
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@p-f p-f

Labels
bug output
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#1043] Fix the Indexed CSV sink. p-f/gradoop
[#1043] Fix the indexed CSV sink. p-f/gradoop
2 participants
@p-f @ChrizZz110