Clear PG_writeback after zil_commit() for sync I/O

When writing via ->writepage() the writeback bit was always cleared
as part of the txg commit callback.  However, when the I/O is also
being written synchronsously to the zil we can immediately clear this
bit.  There is no need to wait for the subsequent TXG sync since the
data is already safe on stable storage.

This has been observed to reduce the msync(2) delay from up to 5
seconds down 10s of miliseconds.  One workload which is expected
to benefit from this are the intermittent samba hands described
in issue openzfs#700.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes openzfs#700
Closes openzfs#907

module/zfs/zfs_vnops.c

@@ -3797,6 +3797,7 @@ zfs_putpage(struct inode *ip, struct page *pp, struct writeback_control *wbc)
 	uint64_t	mtime[2], ctime[2];
 	sa_bulk_attr_t	bulk[3];
 	int		cnt = 0;
+	int		sync;
 
 
 	ASSERT(PageLocked(pp));
@@ -3833,7 +3834,10 @@ zfs_putpage(struct inode *ip, struct page *pp, struct writeback_control *wbc)
 
 	tx = dmu_tx_create(zsb->z_os);
 
-	dmu_tx_callback_register(tx, zfs_putpage_commit_cb, pp);
+	sync = ((zsb->z_os->os_sync == ZFS_SYNC_ALWAYS) ||
+	        (wbc->sync_mode == WB_SYNC_ALL));
+	if (!sync)
+		dmu_tx_callback_register(tx, zfs_putpage_commit_cb, pp);
 
 	dmu_tx_hold_write(tx, zp->z_id, pgoff, pglen);
 
@@ -3862,9 +3866,10 @@ zfs_putpage(struct inode *ip, struct page *pp, struct writeback_control *wbc)
 	dmu_tx_commit(tx);
 	ASSERT3S(err, ==, 0);
 
-	if ((zsb->z_os->os_sync == ZFS_SYNC_ALWAYS) ||
-	    (wbc->sync_mode == WB_SYNC_ALL))
+	if (sync) {
 		zil_commit(zsb->z_log, zp->z_id);
+		zfs_putpage_commit_cb(pp, err);
+	}
 
 	return (err);
 }