Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Colllection of small fixes for splash project (part 2) #1457

Merged
merged 3 commits into from Apr 13, 2018

Conversation

Projects
None yet
6 participants
@garlick
Copy link
Member

garlick commented Apr 12, 2018

This PR contains a couple of items from @trws's #1396 [WIP] Splash PR, and a broker error handling tweak resulting from #1451.

@coveralls

This comment has been minimized.

Copy link

coveralls commented Apr 12, 2018

Coverage Status

Coverage increased (+0.02%) to 79.061% when pulling 5650070 on garlick:hwloc_lite into c2502c4 on flux-framework:master.

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Apr 12, 2018

Codecov Report

Merging #1457 into master will increase coverage by 0.02%.
The diff coverage is 26.31%.

@@            Coverage Diff             @@
##           master    #1457      +/-   ##
==========================================
+ Coverage   78.73%   78.76%   +0.02%     
==========================================
  Files         163      163              
  Lines       30257    30265       +8     
==========================================
+ Hits        23823    23837      +14     
+ Misses       6434     6428       -6
Impacted Files Coverage Δ
src/common/libjsc/jstatctl.c 74.01% <0%> (-0.89%) ⬇️
src/broker/overlay.c 73.81% <50%> (-0.64%) ⬇️
src/broker/broker.c 77.3% <50%> (-0.19%) ⬇️
src/modules/resource-hwloc/resource.c 68.48% <50%> (+0.09%) ⬆️
src/common/libutil/dirwalk.c 93.57% <0%> (-0.72%) ⬇️
src/common/libflux/future.c 88.31% <0%> (-0.47%) ⬇️
src/common/libkvs/kvs_watch.c 89.69% <0%> (-0.43%) ⬇️
src/common/libflux/message.c 81.36% <0%> (+0.23%) ⬆️
src/common/libkvs/kvs_txn.c 75.28% <0%> (+0.56%) ⬆️
... and 3 more
@dongahn

This comment has been minimized.

Copy link
Contributor

dongahn commented Apr 12, 2018

@garlick: Just FYI the scheduler doesn't read rdesc through jsc_query_jcb interface any longer. It now gets that information from the payload of the initial submit event.

If the new interface is needed, I'm not opposed to commit that in, though.

Ultimately, it seems what we will need is "getter" and "setter" interface which can avoid json to native type conversion.

@garlick

This comment has been minimized.

Copy link
Member Author

garlick commented Apr 13, 2018

Ah OK - @trws shall we drop this or do you need it?

@trws

This comment has been minimized.

Copy link
Member

trws commented Apr 13, 2018

Maybe it isn't using it actively right now, but I have at least one version that is using it, it would be best from my perspective to have it if it is reasonable.

@dongahn

This comment has been minimized.

Copy link
Contributor

dongahn commented Apr 13, 2018

Not a problem to fold this in, in that case.

@trws

This comment has been minimized.

Copy link
Member

trws commented Apr 13, 2018

Oh my, thank you for adding the pointer checking @garlick, that was really sloppy of me.

@garlick garlick force-pushed the garlick:hwloc_lite branch 2 times, most recently from e0136e3 to fd58675 Apr 13, 2018

@garlick

This comment has been minimized.

Copy link
Member Author

garlick commented Apr 13, 2018

I updated the commit message on the jsc change to indicate that it is a temporary workaround for splash.

@grondo

This comment has been minimized.

Copy link
Contributor

grondo commented Apr 13, 2018

Restarted a builder that hung after a failure in the cron-faketime tests.

@grondo

This comment has been minimized.

Copy link
Contributor

grondo commented Apr 13, 2018

LGTM, will merge once Travis returns

@grondo

This comment has been minimized.

Copy link
Contributor

grondo commented Apr 13, 2018

I merged @chu11's PR so this needs a rebase, sorry!

@garlick garlick force-pushed the garlick:hwloc_lite branch from fd58675 to 5650070 Apr 13, 2018

garlick added some commits Apr 12, 2018

modules/resource-hwloc: only load info we use
Problem: resource-hwloc module consumes a large amount of
rc1 execution time on a large instance on machine with
complex I/O hierarchy (sierra).

Drop the HWLOC_TOPOLOGY_FLAG_WHOLE_IO flag.

Call hwloc_topology_ignore_type HWLOC_OBJ_CACHE and
HWLOC_OBJ_GROUP to further prune information we don't
currently use.

Fixes #1454
broker: log event redistribution failure
Problem: no error messages are logged when broker
encoutners an error when redistributing events
on the TBON.

Check the return value of functions used to distrubte
event messages, and log any errors.

Fixes #1451
libjsc: add jsc_query_rdesc_efficiently()
Problem: JSON encode/decode of JSC rdesc objects proved
to be a significant bottleneck affecting scheduling
performance for high throughput workloads.

Add jsc_query_rdesc_efficiently() to allow nnodes, ncores,
ntasks, and walltime to be queries directly.

This was solved in a different way in current flux-sched
master, but this is needed as a temporary workaround until
older forks of flux-sched used with the Splash application
catch up with master per @trws.

Fixes #1455

@grondo grondo merged commit 0ebaf83 into flux-framework:master Apr 13, 2018

3 of 4 checks passed

codecov/patch 26.31% of diff hit (target 78.73%)
Details
codecov/project 78.76% (+0.02%) compared to c2502c4
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.02%) to 79.061%
Details
@garlick

This comment has been minimized.

Copy link
Member Author

garlick commented Apr 13, 2018

Thanks for the merge!

@grondo grondo referenced this pull request May 10, 2018

Closed

0.9.0 Release #1479

@garlick garlick deleted the garlick:hwloc_lite branch May 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.