New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add INFO logging saying whether each path.data is on an SSD #10502

Closed
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
6 participants
@mikemccand
Contributor

mikemccand commented Apr 9, 2015

Recently in Lucene we added an IOUtils.spins() method, which returns true for old-fashioned spinning magnets hard drives, and false for SSDs or RAM disks.

It's best effort, but should work well on Linux.

I think we should log this? It can be helpful when looking at a node's log to see whether it's using SSDs for not for it shards...

@kimchy

This comment has been minimized.

Show comment
Hide comment
@kimchy

kimchy Apr 9, 2015

Member

this is going to log it quite a bit for shards allocated and such, maybe we should do it on NodeEnvironment, @s1monw thoughts?

Member

kimchy commented Apr 9, 2015

this is going to log it quite a bit for shards allocated and such, maybe we should do it on NodeEnvironment, @s1monw thoughts?

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Apr 9, 2015

Contributor

yeah lets put it on NodeEnvironmetn for now?

Contributor

s1monw commented Apr 9, 2015

yeah lets put it on NodeEnvironmetn for now?

@mikemccand

This comment has been minimized.

Show comment
Hide comment
@mikemccand

mikemccand Apr 9, 2015

Contributor

OK I'll move to NodeEnvironment...

Contributor

mikemccand commented Apr 9, 2015

OK I'll move to NodeEnvironment...

@mikemccand

This comment has been minimized.

Show comment
Hide comment
@mikemccand

mikemccand Apr 9, 2015

Contributor

OK I moved it to NodeEnvironment. Note that this is not as "accurate" since before we checked the actual Directory instance that Lucene would be using, so if someone is creating some per-index symlink or something then we could log the wrong thing here ... but I agree it is less noisy.

Contributor

mikemccand commented Apr 9, 2015

OK I moved it to NodeEnvironment. Note that this is not as "accurate" since before we checked the actual Directory instance that Lucene would be using, so if someone is creating some per-index symlink or something then we could log the wrong thing here ... but I agree it is less noisy.

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Apr 9, 2015

Contributor

LGTM

Contributor

s1monw commented Apr 9, 2015

LGTM

@mikemccand mikemccand changed the title from Core: add INFO logging saying whether each path.data is on an SSD to Add INFO logging saying whether each path.data is on an SSD Apr 9, 2015

@mikemccand

This comment has been minimized.

Show comment
Hide comment
@mikemccand

mikemccand Apr 9, 2015

Contributor

OK I merged this with the existing (TRACE) logging we were already doing for free space, but upgraded to INFO logging, and added total space, spins, FileStore, FileStore.type() to the output. So now e.g. on my box I see a line like this:

 -> /l/es.logspins/target/J0/./tests-20150409230259-051/0000 has-space/TEST-haswell-CHILD_VM=[0]-CLUSTER_SEED=[2926863498862121027]-HASH=[AFBBDFEB30608]/nodes/0, free_space [260.5gb], usable_space [256.2gb], total_space [465gb], spins? [no], mount [/ (/dev/mapper/haswell--vg-root)], type [btrfs]
Contributor

mikemccand commented Apr 9, 2015

OK I merged this with the existing (TRACE) logging we were already doing for free space, but upgraded to INFO logging, and added total space, spins, FileStore, FileStore.type() to the output. So now e.g. on my box I see a line like this:

 -> /l/es.logspins/target/J0/./tests-20150409230259-051/0000 has-space/TEST-haswell-CHILD_VM=[0]-CLUSTER_SEED=[2926863498862121027]-HASH=[AFBBDFEB30608]/nodes/0, free_space [260.5gb], usable_space [256.2gb], total_space [465gb], spins? [no], mount [/ (/dev/mapper/haswell--vg-root)], type [btrfs]
@rjernst

This comment has been minimized.

Show comment
Hide comment
@rjernst

rjernst Apr 9, 2015

Member

+1

Member

rjernst commented Apr 9, 2015

+1

@rmuir

This comment has been minimized.

Show comment
Hide comment
@rmuir

rmuir Apr 9, 2015

Contributor

looks great. we should try to get them to fix that FileStore bug one day!

Contributor

rmuir commented Apr 9, 2015

looks great. we should try to get them to fix that FileStore bug one day!

@mikemccand mikemccand closed this in 2f5cbf5 Apr 9, 2015

mikemccand added a commit that referenced this pull request Apr 9, 2015

Core: log path.data filesystem details to INFO level
This change logs total space, free space, usable free space, an
estimate of whether the IO system spins (e.g., SSD or not), the mount
point and filesystem type, on node startup.

It produces log output like this:

[2015-04-09 12:09:30,244][INFO ][env                      ] [node_t0] node data locations details:
  -> /l/es.logspins/target/J0/data/TEST-haswell-CHILD_VM=[0]-CLUSTER_SEED=[2926863498862121027]-HASH=[AFC194B1B384B]/nodes/0, free_space [260.6gb], usable_space [256.3gb], total_space [465gb], spins? [no], mount [/ (/dev/mapper/haswell--vg-root)], type [btrfs]

Closes #10502
@mikemccand

This comment has been minimized.

Show comment
Hide comment
@mikemccand

mikemccand Apr 9, 2015

Contributor

On 1.x backport I had to drop the "spins: " part because that's only in Lucene 5.x.

Contributor

mikemccand commented Apr 9, 2015

On 1.x backport I had to drop the "spins: " part because that's only in Lucene 5.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment