Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the functionality for truncating files on Windows #18958

Merged
merged 6 commits into from
May 12, 2023

Conversation

apetenchea
Copy link
Member

@apetenchea apetenchea commented May 11, 2023

Scope & Purpose

This PR addresses an issue caused by the ever growing log files on Windows, due to errors reported from the IOHeartbeatThread. The IOHeartbeatThread feature periodically checks that we are able to write to disk. It does so by creating a file, writing data to it, and finally removing it. In case the file already exists (for example, due to the process being restarted), the IOHeartbeatThread is unable to create the file again on Windows and it gets stuck. Every other second, it prints the following message:

IOHeartbeat: exception when writing test file: write failed for file 'TestFileIOHeartbeat': No error

As you can see, there are two problems:

  1. Reporting the error of the last system call does not work on Windows. Hence, it defaults to No error, which only confuses the user. This PR adds the necessary Windows code for reporting the last system call error. This is an effort to improve the overall error reporting of the system.
  2. The thread gets stuck in this state and produces a log every other second, thus yielding huge log files as time goes by. That was due to the way TRI_createFile used to work, being limited to either creating a non-existing file (CREATE_NEW) or opening an already existing file (OPEN_ALWAYS), but the code for overwriting an existing file was never added (CREATE_ALWAYS). You can think of it as the equivalent of O_TRUNC on Linux, which is exactly how the IOHeartbeatThread creates the file on Linux.
  • 馃挬 Bugfix
  • 馃崟 New feature
  • 馃敟 Performance improvement
  • 馃敤 Refactoring/simplification

Checklist

Related Information

Reported in issue: #18919

@apetenchea apetenchea added this to the devel milestone May 11, 2023
@apetenchea apetenchea self-assigned this May 11, 2023
@cla-bot cla-bot bot added the cla-signed label May 11, 2023
@apetenchea apetenchea changed the title Add the posibility the truncate files in Windows Add the functionality for truncating files on Windows May 11, 2023
lib/Basics/error.cpp Outdated Show resolved Hide resolved
@apetenchea apetenchea force-pushed the bug-fix/issue-18919-ioheartbeat-file-creation branch from 80c0cd5 to 2b0b616 Compare May 12, 2023 08:23
Copy link
Member

@neunhoef neunhoef left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

lib/Basics/error.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@jsteemann jsteemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@KVS85 KVS85 requested a review from Dronplane May 12, 2023 14:23
Copy link
Contributor

@Dronplane Dronplane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@KVS85 KVS85 merged commit 6c0f1f1 into devel May 12, 2023
1 check passed
@KVS85 KVS85 deleted the bug-fix/issue-18919-ioheartbeat-file-creation branch May 12, 2023 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Log file on Windows becomes very large because of TestFileIOHeartbeat
5 participants