FileJournal: fix filejournal alignment

[XinzeChi]

Fixes: #14095
Signed-off-by: Xinze Chi xinze@xsky.com

[liewegas]

Are you sure about this? On Linux the O_DIRECT requirement is actually sector alignment (either 512 or 4096).

[XinzeChi]

@liewegas , I am not sure. There is log in http://tracker.ceph.com/issues/14095
2015-12-17 11:43:19.290709 805215000 -1 FileJournal::_open_any: libaio not compiled in; disabling aio
2015-12-17 11:43:19.635653 805215000 -1 FileJournal::_open_any: libaio not compiled in; disabling aio
2015-12-17 11:43:19.636643 805215000 -1 filestore(testdir/ceph-helpers/0) could not find -1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2015-12-17 11:43:19.651100 805217400 -1 journal block_size 131072 CEPH_MINIMUM_BLOCK_SIZE 4096 CEPH_PAGE_SIZE 4096 header.alignment 131072 bl buffer::list(len=131072, buffer::ptr(0~131072 0x805319000 in raw 0x805319000 len 131072 nref 1)
)

When we use file as journal, it is possible that block_size would be 128k.

block_size = MAX(blksize, (blksize_t)CEPH_MINIMUM_BLOCK_SIZE);

[wjwithagen]

On 22-12-2015 15:18, Chi Xinze wrote:

@liewegas https://github.com/liewegas , I am not sure. There is log in
http://tracker.ceph.com/issues/14095
2015-12-17 11:43:19.290709 805215000 -1 FileJournal::_open_any: libaio
not compiled in; disabling aio
2015-12-17 11:43:19.635653 805215000 -1 FileJournal::_open_any: libaio
not compiled in; disabling aio
2015-12-17 11:43:19.636643 805215000 -1
filestore(testdir/ceph-helpers/0) could not find
-1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2015-12-17 11:43:19.651100 805217400 -1 journal block_size 131072
CEPH_MINIMUM_BLOCK_SIZE 4096 CEPH_PAGE_SIZE 4096 header.alignment 131072
bl buffer::list(len=131072, buffer::ptr(0~131072 0x805319000 in raw
0x805319000 len 131072 nref 1)
)

I do not know why the block_size is 128K in freebsd.

This looks to be the output for the problem with alignments I posted.

It could be because the backing store I'm running this on is ZFS :)
And the basic blocksize on this ZFS filesystem is 128K.
Now 128K would certainly also align on 512B en 4K, but not the other way
around.

The bug/my problem is fixed by Chi Xinze's patch, and compiled in my WIP.

So I'd have to revert, compile and change the blocksize on the
filesystem. Changing blocksize on a ZFS filesystem can be done live and
is valid from the moment it is changed.
I can rebuild the system with the change in FileJournal.cc if needed.
And then run the tests to see what comes of it.

--WjW

[yuyuyu101]

need rebase

[wjwithagen]

On 22-12-2015 15:18, Chi Xinze wrote:

@liewegas https://github.com/liewegas , I am not sure. There is log in
http://tracker.ceph.com/issues/14095
2015-12-17 11:43:19.290709 805215000 -1 FileJournal::_open_any: libaio
not compiled in; disabling aio
2015-12-17 11:43:19.635653 805215000 -1 FileJournal::_open_any: libaio
not compiled in; disabling aio
2015-12-17 11:43:19.636643 805215000 -1
filestore(testdir/ceph-helpers/0) could not find
-1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2015-12-17 11:43:19.651100 805217400 -1 journal block_size 131072
CEPH_MINIMUM_BLOCK_SIZE 4096 CEPH_PAGE_SIZE 4096 header.alignment 131072
bl buffer::list(len=131072, buffer::ptr(0~131072 0x805319000 in raw
0x805319000 len 131072 nref 1)
)

This Error is back.... :( Not sure since when exactly.
Last 4 week I've been redoing my work to get cleaner changes.

But I seem to remember changes being posted to some of the code around
here for
crypto and/or compression.

And yes, on FreeBSD with ZFS the alignment can be 128K

Problem is that I cannot remember what the fix was last time.

The only significat diff I can find with versions I saved:

---

*** 1648,1654 ****
}
// footer
ebl.append((const char_)&h, sizeof(h));
! ebl.rebuild_aligned_size_and_memory(CEPH_MINIMUM_BLOCK_SIZE,
header.alignment);
tbl->claim(ebl);
return h.len;
}
--- 1633,1639 ----
}
// footer
ebl.append((const char_)&h, sizeof(h));
! ebl.rebuild_aligned(CEPH_MINIMUM_BLOCK_SIZE);
tbl->claim(ebl);
return h.len;
}

But I'm not sure that is what disrupted it.

--WjW

[liewegas]

On Fri, 12 Feb 2016, Willem Jan Withagen wrote:

On 22-12-2015 15:18, Chi Xinze wrote:

@liewegas https://github.com/liewegas , I am not sure. There is log in
http://tracker.ceph.com/issues/14095
2015-12-17 11:43:19.290709 805215000 -1 FileJournal::_open_any: libaio
not compiled in; disabling aio
2015-12-17 11:43:19.635653 805215000 -1 FileJournal::_open_any: libaio
not compiled in; disabling aio
2015-12-17 11:43:19.636643 805215000 -1
filestore(testdir/ceph-helpers/0) could not find
-1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2015-12-17 11:43:19.651100 805217400 -1 journal block_size 131072
CEPH_MINIMUM_BLOCK_SIZE 4096 CEPH_PAGE_SIZE 4096 header.alignment 131072
bl buffer::list(len=131072, buffer::ptr(0~131072 0x805319000 in raw
0x805319000 len 131072 nref 1)
)

This Error is back.... :( Not sure since when exactly.
Last 4 week I've been redoing my work to get cleaner changes.

But I seem to remember changes being posted to some of the code around
here for
crypto and/or compression.

And yes, on FreeBSD with ZFS the alignment can be 128K

Problem is that I cannot remember what the fix was last time.

I think the fix is to use a smaller block size for the journal. As long
as the writes aren't horribly efficient this is still fine--I don't think
there is a reason we need to use the exactly block size (i.e. preferred io
size) reported by the fs here.

FWIW we see the same issue on aarch64 with 64k pages.

sage

[wjwithagen]

On 12-2-2016 19:20, Sage Weil wrote:

On Fri, 12 Feb 2016, Willem Jan Withagen wrote:

On 22-12-2015 15:18, Chi Xinze wrote:

@liewegas https://github.com/liewegas , I am not sure. There is log in
http://tracker.ceph.com/issues/14095
2015-12-17 11:43:19.290709 805215000 -1 FileJournal::_open_any: libaio
not compiled in; disabling aio
2015-12-17 11:43:19.635653 805215000 -1 FileJournal::_open_any: libaio
not compiled in; disabling aio
2015-12-17 11:43:19.636643 805215000 -1
filestore(testdir/ceph-helpers/0) could not find
-1/23c2fcde/osd_superblock/0 in index: (2) No such file or directory
2015-12-17 11:43:19.651100 805217400 -1 journal block_size 131072
CEPH_MINIMUM_BLOCK_SIZE 4096 CEPH_PAGE_SIZE 4096 header.alignment 131072
bl buffer::list(len=131072, buffer::ptr(0~131072 0x805319000 in raw
0x805319000 len 131072 nref 1)
)

This Error is back.... :( Not sure since when exactly.
Last 4 week I've been redoing my work to get cleaner changes.

But I seem to remember changes being posted to some of the code around
here for
crypto and/or compression.

And yes, on FreeBSD with ZFS the alignment can be 128K

Problem is that I cannot remember what the fix was last time.

I think the fix is to use a smaller block size for the journal. As long
as the writes aren't horribly efficient this is still fine--I don't think
there is a reason we need to use the exactly block size (i.e. preferred io
size) reported by the fs here.

FWIW we see the same issue on aarch64 with 64k pages.

I saw the pull....
I'll try it in my code, and see if we can still trigger asserts.

There is one re-ordering suggestion in the block with asserts.
That'll give better diagnostics, I'd expect.

--WjW

[XinzeChi]

see #7628