Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAP_SYNC support #2594

Merged
merged 6 commits into from
Feb 9, 2018
Merged

MAP_SYNC support #2594

merged 6 commits into from
Feb 9, 2018

Conversation

plebioda
Copy link
Contributor

@plebioda plebioda commented Feb 7, 2018

This change is Reviewable

PMEM_DEV_DAX, /* device dax */
PMEM_MAP_SYNC, /* mapping with MAP_SYNC flag on dax fs */

MAX_PMEM_TYPE
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add comment here or change the name. Why MAX_*?

Copy link
Member

@pbalcer pbalcer Feb 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are at least two reasons:
1.having a last element of an enum be a MAX_ of something allows you to declare an array with enough space to fit all elements.
2. Every "limit" we declare internally is prefixed with MAX_ so that whenever we need to create an array, iterate over all enum types etc, we can just type MAX_ and let the automatic code completion show us all available options.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@wlemkows
Copy link
Contributor

wlemkows commented Feb 7, 2018

Reviewed 8 of 30 files at r1.
Review status: 8 of 30 files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.


Comments from Reviewable


return 0;
}
case PMEM_MAP_SYNC:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either use braces everywhere or nowhere. This looks particularly jarring because you have 'return ...;' at different indents and at first glance it might look like you are missing a break;

src/common/set.c Outdated
(os_off_t)offset);
}

return util_map_sync(addr, len, prot, flags, part->fd,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens when you call mmap /w MAP_SYNC on devdax?

return util_map_sync(addr, len, prot, flags, part->fd, (os_off_t)offset, part->is_dev_dax ? NULL : map_sync);

src/common/set.c Outdated
for (unsigned p = 1; p < rep->nparts; p++) {
if (map_sync == rep->part[p].map_sync)
continue;
if (rep->part[p].map_sync)
Copy link
Member

@pbalcer pbalcer Feb 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

relying on the previous continue; is confusing.

if (rep->part[p].map_sync != rep->part[p-1].map_sync) {
  ERR("replica ... %s mapped with MAP_SYNC ", ..., rep->part[p].map_sync ? "" : "not";);
}

same below

@@ -2213,7 +2272,7 @@ util_header_create(struct pool_set *set, unsigned repidx, unsigned partidx,
1, POOL_HDR_CSUM_END_OFF);

/* store pool's header */
util_persist_auto(rep->part[partidx].is_dev_dax, hdrp, sizeof(*hdrp));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

util_persist(is_pmem, ...);

the "auto" function when far away from my original intention...

@krzycz
Copy link
Contributor

krzycz commented Feb 7, 2018

Reviewed 25 of 30 files at r1.
Review status: all files reviewed at latest revision, 22 unresolved discussions, some commit checks failed.


src/common/mmap.c, line 187 at r1 (raw file):

	void *base;
	if ((base = util_map(fd, size, MAP_SHARED,
		0, req_align, NULL)) == NULL) {

+TAB


src/common/mmap_posix.c, line 211 at r1 (raw file):

		return ret;
	}

Might be a good idea to put some debug traces here (i.e. with level 4), so you can easily see in the log if this function mapped with MAP_SYNC or not, and why.


src/common/mmap_windows.c, line 145 at r1 (raw file):

		addr, len, proto, flags, fd, offset);

	*map_sync = 0;

I think we could refactor the code somehow and check if the file pointed by "fd" is on a dax-enabled filesystem. On Windows, this is enough to treat it as "pmem" (like for MAP_SYNC).
IOW, we could pass MAP_SYNC to Windows implementation of mmap() and handle it there by checking the volume properties. If it supports DAX, it means it supports MAP_SYNC.
Thanks to that, there would be one code path for Linux and Windows.


src/common/set.c, line 351 at r1 (raw file):

/*
 * util_map_part_type -- memory map part file into memory according

"memory map into memory" sounds like tautology


src/common/set.c, line 355 at r1 (raw file):

 */
static void *
util_map_part_type(struct pool_set_part *part, void *addr, size_t len,

Is this needed? Why not to map everything with util_map_sync()?
I guess it will return map_sync=0 for device dax, won't it?


src/common/set.c, line 467 at r1 (raw file):

	int prot = rdonly ? PROT_READ : PROT_READ | PROT_WRITE;
	void *addrp = util_map_part_type(part, addr, size, prot, flags,
			offset, &part->map_sync);

No check for MAP_FAILED?


src/common/set.c, line 1103 at r1 (raw file):

			continue;
		if (rep->part[p].map_sync)
			ERR("replica #%u part %u mapped with MAP_SYNC",

Is it really an error? Why not to treat the entire replica as non-pmem, if any part/hdr was mapped without MAP_SYNC?


src/common/set.c, line 2496 at r1 (raw file):

 *
 * It's enough to check only first part because it's already verified
 * that either all or none parts are deivce dax or mapped with MAP_SYNC.

device


src/common/set.c, line 2499 at r1 (raw file):

 */
static void
util_replica_set_is_pmem(struct pool_replica *rep)

I don't like this one-liner function that is only used twice. "inline" maybe?


src/common/set.h, line 104 at r1 (raw file):

	void *hdr;		/* base address of header */
	size_t hdrsize;		/* size of the header mapping */
	int hdr_map_sync;	/* header mapped with MAP_SYNC */

Does it make sense to keep this info per part/hdr? I think it's enough to have it per replica.


src/test/RUNTESTS, line 445 at r1 (raw file):

PMEMDETECT="tools/pmemdetect/pmemdetect.static-nondebug"

if [ "$PMEM_FS_DIR_FORCE_PMEM" = "1" ]; then

What about moving all the PMEM/non-PMEM detection here, instead or running pmemdetect for each and every test in require_fs_type?


src/test/testconfig.sh.example, line 45 at r1 (raw file):

# tests that require pmem.
#
# If you don't have real PMEM or the filesystem does not support MAP_SYNC flag,

It somehow duplicates the sentence above. They could be merged into one, I believe.


src/test/tools/pmemdetect/pmemdetect.c, line 155 at r1 (raw file):

	if (ret) {
		/* no such file */

If the file does not exists, it's the same case as testing directory with tmpfile.


src/test/tools/pmemdetect/pmemdetect.c, line 163 at r1 (raw file):

	} else {
		/* file exist */
		*size = (size_t)buf.st_size;

why not 0 ?


src/test/tools/pmemdetect/pmemdetect.c, line 233 at r1 (raw file):

 */
static int
supports_map_sync(const char *path)

Perhaps I missing something, but it appears to me that checking for MAP_SYNC is actually the same as checking for "is_pmem", isn't it? The only difference is that pmem_map_file is affected by PMEM_IS_PMEM_FORCE, but we can unset it for a moment (saving the previous value), run pmemdetect and restore env vars.


src/test/unittest/unittest.ps1, line 1177 at r1 (raw file):

        'pmem' {
            sv -Name DIR ($Env:PMEM_FS_DIR + $tail)
            if ($Env:PMEM_FS_DIR_FORCE_PMEM -eq "1" -or $Env:PMEM_FS_DIR_FORCE_PMEM -eq "2") {

Does it apply to windows?


src/test/unittest/unittest.ps1, line 1188 at r1 (raw file):

                sv -Name DIR ($Env:PMEM_FS_DIR + $tail)
                $Global:REAL_FS='pmem'
		if ($Env:PMEM_FS_DIR_FORCE_PMEM -eq "1" -or $Env:PMEM_FS_DIR_FORCE_PMEM -eq "2") {

indent with spaces, pls - it's powershell


src/tools/pmempool/info.c, line 500 at r1 (raw file):

		case 'p':
		{
			errno = 0;

it is set to 0 three lines below
now, saving old errno does not make any sense


Comments from Reviewable

@krzycz
Copy link
Contributor

krzycz commented Feb 7, 2018

Review status: all files reviewed at latest revision, 22 unresolved discussions, some commit checks failed.


src/common/set.c, line 364 at r1 (raw file):

Previously, pbalcer (Piotr Balcer) wrote…

what happens when you call mmap /w MAP_SYNC on devdax?

return util_map_sync(addr, len, prot, flags, part->fd, (os_off_t)offset, part->is_dev_dax ? NULL : map_sync);

Good question. I believe it fails, as MAP_SYNC is a feature of a filesystem, not a device.
However, it shouldn't be a problem if MAP_SYNC is supported (ignored) by device dax, as - by definition - device dax provides the same consistency guarantees as MAP_SYNC.


Comments from Reviewable

@plebioda
Copy link
Contributor Author

plebioda commented Feb 7, 2018

Review status: all files reviewed at latest revision, 22 unresolved discussions, some commit checks failed.


src/common/mmap.c, line 187 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

+TAB

Done.


src/common/mmap_posix.c, line 211 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Might be a good idea to put some debug traces here (i.e. with level 4), so you can easily see in the log if this function mapped with MAP_SYNC or not, and why.

Done.


src/common/mmap_windows.c, line 145 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

I think we could refactor the code somehow and check if the file pointed by "fd" is on a dax-enabled filesystem. On Windows, this is enough to treat it as "pmem" (like for MAP_SYNC).
IOW, we could pass MAP_SYNC to Windows implementation of mmap() and handle it there by checking the volume properties. If it supports DAX, it means it supports MAP_SYNC.
Thanks to that, there would be one code path for Linux and Windows.

I was thinking about this. There is even such function - util_fd_is_device_dax. But this is once again prepending Linux specific feature on windows and I don't like such approach. I'm aware this is not the ideal implementation but I decided to go this way because I believe it is the easiest way to implement it.


src/common/os_deep_persist_linux.c, line 104 at r1 (raw file):

Previously, pbalcer (Piotr Balcer) wrote…

either use braces everywhere or nowhere. This looks particularly jarring because you have 'return ...;' at different indents and at first glance it might look like you are missing a break;

Done.


src/common/set.c, line 351 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

"memory map into memory" sounds like tautology

Done.


src/common/set.c, line 355 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Is this needed? Why not to map everything with util_map_sync()?
I guess it will return map_sync=0 for device dax, won't it?

Done.


src/common/set.c, line 364 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Good question. I believe it fails, as MAP_SYNC is a feature of a filesystem, not a device.
However, it shouldn't be a problem if MAP_SYNC is supported (ignored) by device dax, as - by definition - device dax provides the same consistency guarantees as MAP_SYNC.

Done.


src/common/set.c, line 467 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

No check for MAP_FAILED?

Done.


src/common/set.c, line 1102 at r1 (raw file):

Previously, pbalcer (Piotr Balcer) wrote…

relying on the previous continue; is confusing.

if (rep->part[p].map_sync != rep->part[p-1].map_sync) {
  ERR("replica ... %s mapped with MAP_SYNC ", ..., rep->part[p].map_sync ? "" : "not";);
}

same below

Done.


src/common/set.c, line 1103 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Is it really an error? Why not to treat the entire replica as non-pmem, if any part/hdr was mapped without MAP_SYNC?

We have two options. Report an error or silently fallback to non-pmem and silently degradate performance significantly. I think we discussed this offline that we should fail in such case - it works the same way right now with device dax. I think this option is better but I'm ok with changing this.


src/common/set.c, line 2496 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

device

Done.


src/common/set.c, line 2499 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

I don't like this one-liner function that is only used twice. "inline" maybe?

one-line of code but multiple lines of comment. I believe it is worht keeping this in one place because it is really important line of code.


src/common/set.h, line 104 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Does it make sense to keep this info per part/hdr? I think it's enough to have it per replica.

The header is mmapped in util_map_hdr function which takes a pool_set_part structure. I found this structure convenient to keep this information - I need to store this flag somewhere. After mapping all headers we check if all values are the same.


src/test/RUNTESTS, line 445 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

What about moving all the PMEM/non-PMEM detection here, instead or running pmemdetect for each and every test in require_fs_type?

I don't think I understand your comment. The pmemdetect is executed only once (below).


src/test/testconfig.sh.example, line 45 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

It somehow duplicates the sentence above. They could be merged into one, I believe.

Done.


src/test/tools/pmemdetect/pmemdetect.c, line 155 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

If the file does not exists, it's the same case as testing directory with tmpfile.

Yes. But for directory we set PMEM_FILE_TMPFILE.


src/test/tools/pmemdetect/pmemdetect.c, line 163 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

why not 0 ?

Done.


src/test/tools/pmemdetect/pmemdetect.c, line 233 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Perhaps I missing something, but it appears to me that checking for MAP_SYNC is actually the same as checking for "is_pmem", isn't it? The only difference is that pmem_map_file is affected by PMEM_IS_PMEM_FORCE, but we can unset it for a moment (saving the previous value), run pmemdetect and restore env vars.

yes and no. supports_map_sync returns true only for files on FS which do support MAP_SYNC. is_pmem returns true for MAP_SYNC FSs, device dax and/or PMEM_IS_PMEM_FORCE. I found such functionality useful for testing environment so I implemented it. IMHO it is cleaner to call this function instead of playing with environment variables - this is fragile.


src/test/unittest/unittest.ps1, line 1177 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Does it apply to windows?

Done.


src/test/unittest/unittest.ps1, line 1188 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

indent with spaces, pls - it's powershell

Done.


src/tools/pmempool/info.c, line 500 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

it is set to 0 three lines below
now, saving old errno does not make any sense

Mistake when resolving confict


Comments from Reviewable

@plebioda plebioda force-pushed the map-sync branch 3 times, most recently from 9c5e331 to e29b39b Compare February 7, 2018 15:37
@krzycz
Copy link
Contributor

krzycz commented Feb 7, 2018

Reviewed 4 of 28 files at r2, 24 of 24 files at r3.
Review status: 27 of 28 files reviewed at latest revision, 17 unresolved discussions, some commit checks broke.


src/common/mmap_posix.c, line 193 at r3 (raw file):

	os_off_t offset, int *map_sync)
{
	LOG(15, "addr %p len %zu proto %x flags %x fd %d offset %ld "

BTW, it should be "%lld" IMHO.


src/common/mmap_posix.c, line 208 at r3 (raw file):

			fd, offset);
	if (ret != MAP_FAILED) {
		LOG(4, "mmap with MAP_SYNC succeeded");

+errno
EINVAL or ENOTSUP - old kernel / MAP_SYNC not supported by this filesystem


src/common/mmap_windows.c, line 145 at r1 (raw file):

Previously, plebioda (Pawel Lebioda) wrote…

I was thinking about this. There is even such function - util_fd_is_device_dax. But this is once again prepending Linux specific feature on windows and I don't like such approach. I'm aware this is not the ideal implementation but I decided to go this way because I believe it is the easiest way to implement it.

Let's leave it for a post-merge refactoring.


src/common/os_deep_persist_linux.c, line 88 at r3 (raw file):

os_deep_persist_type(const struct map_tracker *mt, void *addr, size_t len)
{
	ASSERTne(mt->type, MAX_PMEM_TYPE);

LOG(15, ...)


src/common/os_deep_persist_linux.c, line 88 at r3 (raw file):

os_deep_persist_type(const struct map_tracker *mt, void *addr, size_t len)
{
	ASSERTne(mt->type, MAX_PMEM_TYPE);

ASSERT(mt->type < MAX_PMEM_TYPE);
Also, it duplicates ASSERT in switch default.


src/common/os_deep_persist_linux.c, line 92 at r3 (raw file):

	switch (mt->type) {
	case PMEM_DEV_DAX:
		LOG(15, "pmem_persist addr %p, len %lu", addr, len);

pmem_persist() has it's own debug traces. this seems to be unnecessary


src/common/set.c, line 467 at r1 (raw file):

Previously, plebioda (Pawel Lebioda) wrote…

Done.

Nope. Please, revert the original check here. The logic was different.
NOTE: the check was on addrp not addr.


src/common/set.c, line 1103 at r1 (raw file):

Previously, plebioda (Pawel Lebioda) wrote…

We have two options. Report an error or silently fallback to non-pmem and silently degradate performance significantly. I think we discussed this offline that we should fail in such case - it works the same way right now with device dax. I think this option is better but I'm ok with changing this.

OK. Just remember to update documentation.


src/libpmem/pmem_posix.c, line 70 at r3 (raw file):

pmem_map_register(int fd, size_t len, const char *path, int is_dev_dax)
{
	void *addr;

LOG(3, ...)


src/libpmem/pmem_posix.c, line 72 at r3 (raw file):

	void *addr;
	if (is_dev_dax) {
		if ((addr = util_map(fd, len, MAP_SHARED, 0, 0, NULL)) == NULL)

I think we could cal util_map() with the same args for any file (including devdax).
If it succeeds, then depending on map_sync and is_dev_dax, the proper type should be passed to util_range_register.


src/test/RUNTESTS, line 445 at r1 (raw file):

Previously, plebioda (Pawel Lebioda) wrote…

I don't think I understand your comment. The pmemdetect is executed only once (below).

See unittest.sh.


src/test/tools/pmemdetect/pmemdetect.c, line 155 at r1 (raw file):

Previously, plebioda (Pawel Lebioda) wrote…

Yes. But for directory we set PMEM_FILE_TMPFILE.

I know. But if the file does not exists, it is created and unlinked immediately after the check, so... it acts as a temporary file - only the name is different.


src/test/tools/pmemdetect/pmemdetect.c, line 233 at r1 (raw file):

Previously, plebioda (Pawel Lebioda) wrote…

yes and no. supports_map_sync returns true only for files on FS which do support MAP_SYNC. is_pmem returns true for MAP_SYNC FSs, device dax and/or PMEM_IS_PMEM_FORCE. I found such functionality useful for testing environment so I implemented it. IMHO it is cleaner to call this function instead of playing with environment variables - this is fragile.

We never use pmemdetect to check if device dax is pmem, so this is not the case.
However, I agree that playing with envvars is hairy, so let's leave it as is.


Comments from Reviewable

@plebioda plebioda force-pushed the map-sync branch 5 times, most recently from e613eb3 to 7584ff5 Compare February 8, 2018 11:38
@codecov-io
Copy link

codecov-io commented Feb 8, 2018

Codecov Report

Merging #2594 into master will decrease coverage by 0.06%.
The diff coverage is 62.83%.

@@            Coverage Diff             @@
##           master    #2594      +/-   ##
==========================================
- Coverage   80.39%   80.32%   -0.07%     
==========================================
  Files         146      146              
  Lines       22367    22442      +75     
==========================================
+ Hits        17981    18027      +46     
- Misses       4386     4415      +29

If PMEM_FS_DIR_FORCE_PMEM is set to 1 the pmem_is_pmem will not check
if the mapping has been created with MAP_SYNC. This patch reports an
error if this variable is set to 1 AND the PMEM_FS_DIR supports
MAP_SYNC.
@krzycz
Copy link
Contributor

krzycz commented Feb 8, 2018

Reviewed 7 of 7 files at r4.
Review status: all files reviewed at latest revision, 14 unresolved discussions.


Comments from Reviewable

Do not check if PMEM_FS_DIR is pmem and NON_PMEM_FS_DIR is not pmem for
evey single require_fs_type execution but once per RUNTESTS[.PS1]. This
couse the pmemdetect is invoked less frequently and should positively
impact overall test execution time.
@plebioda
Copy link
Contributor Author

plebioda commented Feb 9, 2018

Review status: 22 of 30 files reviewed at latest revision, 14 unresolved discussions.


src/common/mmap_posix.c, line 193 at r3 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

BTW, it should be "%lld" IMHO.

../../src/../src/common/mmap_posix.c:193:10: error: format ‘%lld’ expects argument of type ‘long long int’, but argument 11 has type ‘os_off_t {aka long int}’ [-Werror=format=]

src/common/mmap_posix.c, line 208 at r3 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

+errno
EINVAL or ENOTSUP - old kernel / MAP_SYNC not supported by this filesystem

Done.


src/common/mmap_windows.c, line 145 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Let's leave it for a post-merge refactoring.

Done.


src/common/os_deep_persist_linux.c, line 88 at r3 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

LOG(15, ...)

Done.


src/common/os_deep_persist_linux.c, line 88 at r3 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

ASSERT(mt->type < MAX_PMEM_TYPE);
Also, it duplicates ASSERT in switch default.

Done.


src/common/os_deep_persist_linux.c, line 92 at r3 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

pmem_persist() has it's own debug traces. this seems to be unnecessary

Done.


src/common/set.c, line 467 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

Nope. Please, revert the original check here. The logic was different.
NOTE: the check was on addrp not addr.

Done.


src/libpmem/pmem_posix.c, line 70 at r3 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

LOG(3, ...)

Done.


src/libpmem/pmem_posix.c, line 72 at r3 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

I think we could cal util_map() with the same args for any file (including devdax).
If it succeeds, then depending on map_sync and is_dev_dax, the proper type should be passed to util_range_register.

Done.


src/test/RUNTESTS, line 445 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

See unittest.sh.

Done.


src/test/tools/pmemdetect/pmemdetect.c, line 155 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

I know. But if the file does not exists, it is created and unlinked immediately after the check, so... it acts as a temporary file - only the name is different.

So what do you propose ? Ignore the last part of path and create temporary file ? Current implementation is simple and clear - if you provide a path to a file which doesn't exist you would like to check "if I create such a file will it be a pmem or not ?"


src/test/tools/pmemdetect/pmemdetect.c, line 233 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

We never use pmemdetect to check if device dax is pmem, so this is not the case.
However, I agree that playing with envvars is hairy, so let's leave it as is.

Done.


Comments from Reviewable

@krzycz
Copy link
Contributor

krzycz commented Feb 9, 2018

Reviewed 8 of 8 files at r5.
Review status: all files reviewed at latest revision, 2 unresolved discussions.


Comments from Reviewable

@krzycz
Copy link
Contributor

krzycz commented Feb 9, 2018

:lgtm:


Review status: all files reviewed at latest revision, 2 unresolved discussions.


Comments from Reviewable

@krzycz krzycz merged commit 0b7f54c into pmem:master Feb 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants