Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hariharan/vbucket trait #64

Merged

Conversation

hariharan-devarajan
Copy link
Collaborator

This PR consists of functionality required for POSIX and STDIO adapter. Currently, It doesn't support shared traits/vBuckets as none of the metadata is stored in shared memory. I believe we will do it as we need it.

@ChristopherHogan
Copy link
Collaborator

Could you please remove the *.pyc files and add __pycache__/ to .gitignore?

@ChristopherHogan
Copy link
Collaborator

There are a lot of arbitrary white space changes. They really clutter up the PR and make it difficult to focus on the important changes.

@hariharan-devarajan
Copy link
Collaborator Author

I have tried to reduce the whitespace a little. Hope that helps.

@ChristopherHogan
Copy link
Collaborator

Thank you.

@ChristopherHogan
Copy link
Collaborator

This PR triples the compilation time from ~7 seconds to ~20 seconds. Is there anything you can do to lower that?

@hariharan-devarajan
Copy link
Collaborator Author

The compilation unit for Hermes is the same as you recommended. It must be due to the test case which uses catch2 framework.

@ChristopherHogan
Copy link
Collaborator

ChristopherHogan commented Jan 14, 2021

The compilation unit for Hermes is the same as you recommended.

Actually, I recommended that you #include traits.cc in buffer_pool.cc to keep the number of compilation units low, but instead you have traits.cc in HERMES_SRCS in src/CMakeLists.txt, which creates an additional compilation unit. Making that change appears to reduce the time by a few seconds.

Catch2 compiling that slow is a deal breaker for me. Can you see if this helps?

src/api/traits.h Outdated
std::string blob_name;
};
typedef BlobInfo TraitInput;
// typedef void* TraitCallback;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove commented out code, or add a comment explaining why it's left in.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

src/api/traits.h Outdated

struct Trait;
typedef std::function<void(TraitInput &, Trait *)> TraitCallback;
// typedef void(*TraitCallback)(TraitInput &, Trait *);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove commented out code, or add a comment explaining why it's left in.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

src/api/traits.h Outdated
Trait(TraitID id, TraitIdArray conflict_traits, TraitType type);
};

#define FILE_TRAIT 10
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our convention is to prefix preprocessor symbols with HERMES_.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@hariharan-devarajan
Copy link
Collaborator Author

#include traits.cc

The compilation unit for Hermes is the same as you recommended.

Actually, I recommended that you #include traits.cc in buffer_pool.cc to keep the number of compilation units low, but instead you have traits.cc in HERMES_SRCS in src/CMakeLists.txt, which creates an additional compilation unit. Making that change appears to reduce the time by a few seconds.

Catch2 compiling that slow is a deal breaker for me. Can you see if this helps?

I have added both changes. The reason for keeping trait.cc there was because we have vBucket there as well.

else
std::cerr << "VBucket " << initial_name << " exists\n";
(void)ctx;
if (IsBucketNameTooLong(name_)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be IsVBucketNameTooLong?.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FIXED

VBucketID result = GetVBucketIdByName(context, rpc, name.c_str());

if (result.as_int != 0) {
LOG(INFO) << "Opening Bucket '" << name << "'" << std::endl;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Opening VBucket"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FIXED

LOG(INFO) << "Opening Bucket '" << name << "'" << std::endl;
IncrementRefcount(context, rpc, result);
} else {
LOG(INFO) << "Creating Bucket '" << name << "'" << std::endl;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Creating VBucket"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FIXED

@ChristopherHogan
Copy link
Collaborator

Unfortunately CATCH_CONFIG_FAST_COMPILE doesn't appear to make any difference.

@ChristopherHogan
Copy link
Collaborator

Thanks for the changes. I'll merge this so you can make progress on the adapter, but I think we should discuss alternatives to Catch2. I'm not willing to triple our compilation times.

@ChristopherHogan ChristopherHogan merged commit 71ccd1b into HDFGroup:master Jan 14, 2021
@hariharan-devarajan
Copy link
Collaborator Author

Thanks for the changes. I'll merge this so you can make progress on the adapter, but I think we should discuss alternatives to Catch2. I'm not willing to triple our compilation times.

Sure, we can search more if we can optimize this. Or need to switch libraries.

ChristopherHogan added a commit that referenced this pull request Jan 15, 2021
@hariharan-devarajan hariharan-devarajan deleted the hariharan/vbucket_trait branch March 4, 2021 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants