Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow multiple named shared memory regions #4982

Merged
merged 7 commits into from
Apr 5, 2018
Merged

Allow multiple named shared memory regions #4982

merged 7 commits into from
Apr 5, 2018

Conversation

TheMarex
Copy link
Member

@TheMarex TheMarex commented Mar 29, 2018 •
edited

Issue

This PR adds support for having multiple named shared memory regions.
This is going to unlock #1221 and #4873 by making it easier to have multiple named regions.

Tasklist

Requirements / Relations

Based on #4975.

@TheMarex TheMarex force-pushed the feature/multiple_shm branch 4 times, most recently from 982c712 to 410f545 Compare April 4, 2018 13:23
@miccolis miccolis requested a review from oxidase April 4, 2018 15:39
oxidase
oxidase approved these changes Apr 5, 2018
View reviewed changes
Copy link
Contributor

@oxidase oxidase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Just small comments about output

src/storage/storage.cpp Outdated
util::UnbufferedLog() << "ok.";
}

util::Log() << "Loading data into " << regionToString(next_region);
util::Log() << "Loading data into " << shm_key;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shm_key has 8 bits type and will be printed as a character Loading data into �. static_cast<int> or (int)?

src/storage/storage.cpp Outdated
}

util::Log() << "All data loaded. Notify all client about new data in "
<< regionToString(next_region) << " with timestamp " << next_timestamp;
util::Log() << "All data loaded. Notify all client about new data in " << shm_key
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

src/storage/storage.cpp Outdated
{
util::UnbufferedLog() << "Marking old shared memory region "
<< regionToString(in_use_region) << " for removal... ";
util::UnbufferedLog() << "Marking old shared memory region " << in_use_key
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

src/tools/store.cpp Outdated
{
auto id = shared_register.Find(name);
auto region = shared_register.GetRegion(id);
osrm::util::Log() << name << "\t" << (int)region.shm_key << "\t" << region.timestamp;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it make sense to add "\tsize" column?

        auto shm = osrm::storage::makeSharedMemory(region.shm_key);
        osrm::util::Log() << name << "\t" << (int)region.shm_key << "\t" << region.timestamp << "\t" << shm->Size();

otherwise an extra ipcs -m call is needed

include/storage/shared_datatype.hpp Outdated
}
}

void Deregister(const RegionID key) { regions[key] = SharedRegion{}; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method is not used and DataWatchdogImpl assumes that the shared region pointer won't be changed during an engine process life-time. Is a follow-up issue about deleting named regions needed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I wanted to use this for the spring-clean command, but instead we just overwrite the whole registry.

src/osrm/osrm.cpp
@@ -28,12 +28,6 @@ OSRM::OSRM(engine::EngineConfig &config)
throw util::exception("Required files are missing, cannot continue. Have all the "
"pre-processing steps been run?");
}
else if (config.use_shared_memory)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@oxidase
Copy link
Contributor

oxidase commented Apr 5, 2018

It is also possible to add features/options/datastore/datastore.feature options tests to increase coverage

@datastore @options @help
Feature: osrm-datastore command line options: list

    Background:
        Given the profile "testbot"
        And the node map
            """
            a b
            """
        And the ways
            | nodes |
            | ab    |
        And the data has been contracted

    Scenario: osrm-datastore - Help should be shown when no options are passed
        When I try to run "osrm-datastore --dataset-name test_dataset_42 {processed_file}"
        Then it should exit successfully
        When I try to run "osrm-datastore --list"
        Then it should exit successfully
        And stdout should contain "test_dataset_42/data"

@TheMarex
Copy link
Member Author

TheMarex commented Apr 5, 2018

@oxidase thanks for the test case! Changed all locations and added the test. Should be good to merge after Travis passes.

@TheMarex TheMarex mentioned this pull request Apr 5, 2018
Merged
5 tasks
@TheMarex TheMarex merged commit c7daa52 into master Apr 5, 2018
@TheMarex TheMarex deleted the feature/multiple_shm branch April 5, 2018 13:12
@TheMarex TheMarex mentioned this pull request Apr 6, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oxidase oxidase oxidase approved these changes

Assignees
No one assigned
Labels
Ready To Merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@TheMarex @oxidase