Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add max file size bootstrap check #25974

Merged
merged 4 commits into from Jul 31, 2017

Conversation

Projects
None yet
5 participants
@jasontedor
Copy link
Member

commented Jul 31, 2017

This commit adds a bootstrap check for the maximum file size, and ensures the limit is set correctly when Elasticsearch is installed as a service on systemd-based systems.

Add max file size bootstrap check
This commit adds a bootstrap check for the maximum file size, and
ensures the limit is set correctly when Elasticsearch is installed as a
service on systemd-based systems.
@jasontedor

This comment has been minimized.

Copy link
Member Author

commented Jul 31, 2017

Otherwise this can happen:

java.io.IOException: File too large
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) ~[?:?]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:?]
	at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:?]
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) ~[?:?]
	at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) ~[?:1.8.0_141]
	at java.nio.channels.Channels.writeFully(Channels.java:101) ~[?:1.8.0_141]
	at java.nio.channels.Channels.access$000(Channels.java:61) ~[?:1.8.0_141]
	at java.nio.channels.Channels$1.write(Channels.java:174) ~[?:1.8.0_141]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_141]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_141]
	at org.elasticsearch.index.translog.TranslogWriter$BufferedChannelOutputStream.flush(TranslogWriter.java:450) ~[elasticsearch-6.0.0-beta1-SNAPSHOT.jar:6.0.0-beta1-SNAPSHOT]
@@ -40,6 +40,7 @@
public static final int ENOMEM = 12;
public static final int RLIMIT_MEMLOCK = Constants.MAC_OS_X ? 6 : 8;
public static final int RLIMIT_AS = Constants.MAC_OS_X ? 5 : 9;
public static final int RLIMIT_FSIZE = Constants.MAC_OS_X ? 1 : 1;

This comment has been minimized.

Copy link
@jpountz

jpountz Jul 31, 2017

Contributor

the ternary operator looks useless?

This comment has been minimized.

Copy link
@jasontedor

jasontedor Jul 31, 2017

Author Member

I prefer it this way because it's consistent with how the other RLIMIT constants are defined where there are differences between macOS and Linux.


The segment files that are the components of individual shards and the translog
generations that are components of the translog can get large (on the order of
hundreds of megabytes and even extending into gigabytes). On systems where the

This comment has been minimized.

Copy link
@jpountz

jpountz Jul 31, 2017

Contributor

the phrasing ("even") suggests having files that are 1GB is uncommon, while I think it is actually common?

This comment has been minimized.

Copy link
@jasontedor

jasontedor Jul 31, 2017

Author Member

You're right, that was not intended, I pushed ee35808.

public void testSetMaxFileSize() throws IOException {
if (Constants.LINUX) {
final List<String> lines = Files.readAllLines(PathUtils.get("/proc/self/limits"));
if (!lines.isEmpty()) {

This comment has been minimized.

Copy link
@jpountz

jpountz Jul 31, 2017

Contributor

this if statement looks useless?

This comment has been minimized.

Copy link
@jasontedor

jasontedor Jul 31, 2017

Author Member

Inside this block we make assertions and return from the method. Otherwise, there is a fail below this block that fails the test because we can not assert anything if for some reason we did not read any content from /proc/self/limits

This comment has been minimized.

Copy link
@jpountz

jpountz Jul 31, 2017

Contributor

the for loop will prevent from reaching the assertions and returning if lines is empty?

This comment has been minimized.

Copy link
@jasontedor

jasontedor Jul 31, 2017

Author Member

You're right, I pushed 34a82d5.

@jasontedor

This comment has been minimized.

Copy link
Member Author

commented Jul 31, 2017

test this please

jasontedor added some commits Jul 31, 2017

@jasontedor jasontedor merged commit 2ef0f8a into elastic:master Jul 31, 2017

1 of 2 checks passed

elasticsearch-ci Build started sha1 is merged.
Details
CLA Commit author is a member of Elasticsearch
Details

jasontedor added a commit that referenced this pull request Jul 31, 2017

Add max file size bootstrap check
This commit adds a bootstrap check for the maximum file size, and
ensures the limit is set correctly when Elasticsearch is installed as a
service on systemd-based systems.

Relates #25974

jasontedor added a commit that referenced this pull request Jul 31, 2017

Add max file size bootstrap check
This commit adds a bootstrap check for the maximum file size, and
ensures the limit is set correctly when Elasticsearch is installed as a
service on systemd-based systems.

Relates #25974

jasontedor added a commit that referenced this pull request Jul 31, 2017

Add max file size bootstrap check
This commit adds a bootstrap check for the maximum file size, and
ensures the limit is set correctly when Elasticsearch is installed as a
service on systemd-based systems.

Relates #25974
@jasontedor

This comment has been minimized.

Copy link
Member Author

commented Jul 31, 2017

Thank you for the review @jpountz.

@jasontedor jasontedor deleted the jasontedor:max-file-size-check branch Jul 31, 2017

@colings86 colings86 added v6.0.0-beta1 and removed v6.0.0 labels Aug 3, 2017

@lcawl lcawl removed the v6.1.0 label Dec 12, 2017

@jimczi jimczi added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.