To avoid complex lock ordering issues loop_ctl_mutex should not
be held over add_disk.  Add a new Lo_new state for a loop device
that has just been created but which is not live yet.

Reported-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Signed-off-by: Christoph Hellwig <hch@lst.de>

To avoid complex lock ordering issues always delete loop devices outside
of loop_ctl_mutex.  In loop_control_remove the Lo_deleting state can
be used to prevent further lookups, and given that module unload is
synchronized vs new opens of the control device and thus ioctls there
is no need for locks there at all.

Based on patches from Hillf Danton <hdanton@sina.com> and
Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>.

Reported-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Signed-off-by: Christoph Hellwig <hch@lst.de>

Signed-off-by: Christoph Hellwig <hch@lst.de>

Besides the already special cased non-transform fastpath there are only
two different transfers.  Hardcode them instead of the maze of indirect
calls and switch the whole cryptoloop code to use IS_ENABLED for dead
code elimination.

Signed-off-by: Christoph Hellwig <hch@lst.de>

No need to keep a separate loadable module infrastructure for a tiny
amount of cryptoapi glue, especially as unloading of the cryptoloop
module leads to nasty interactions with the loop device state machine
through loop_unregister_transfer.

Signed-off-by: Christoph Hellwig <hch@lst.de>

Returning an error here is no useful.  So remove the cargo culted check
in cryptoloop and remove the return value.

Signed-off-by: Christoph Hellwig <hch@lst.de>

Never set to anything useful.

Signed-off-by: Christoph Hellwig <hch@lst.de>

* io_uring-5.14:
  io_uring: pin ctx on fallback execution
  io_uring: only assign io_uring_enter() SQPOLL error in actual error case
  tools/io_uring/io_uring-cp: sync with liburing example

Signed-off-by: Jens Axboe <axboe@kernel.dk>

* for-5.15/io_uring:
  io_uring: improve same wq polling
  io_uring: improve tctx_task_work() ctx referencing
  io_uring: reuse io_req_complete_post()
  io_uring: better encapsulate buffer select for rw

Move earlier the check for whether __io_queue_proc() tries to poll
already polled waitqueue, and do the same for the second poll entry, if
any. Shouldn't really matter, but at least it would have a more
predictable behaviour.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/8cb428cfe8ade0fd055859fabb878db8777d4c2f.1629228203.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

io_uring processed by tctx_task_work() can't get freed until the
function returns. The reason is that io_ring_exit_work() executes a
task_work after all references are put, where the task works
execution is naturally serialised. Remove extra ctx pinning.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/8494d5bac9e30dba7928d33b18f4d0ddeb9899ce.1629228203.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

We have io_req_complete_post() to post a CQE and put the request. It
takes care of all synchronisation and is more concise and efficent, so
replace all hancoded occurrences of
"lock; post CQE; unlock; + put_req()" with io_req_complete_post().

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/2c83463458a613f9d870e5147eb134da2aa70779.1629228203.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Make io_put_rw_kbuf() to do the REQ_F_BUFFER_SELECTED check, so all the
callers don't need to hand code it. The number of places where we call
io_put_rw_kbuf() is growing, so saves some pain.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/3df3919e5e7efe03420c44ab4d9317a81a9cf398.1629228203.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Pin ring in io_fallback_req_func() by briefly elevating ctx->refs in
case any task_work handler touches ctx after releasing a request.

Fixes: 9011bf9 ("io_uring: fix stuck fallback reqs")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/833a494713d235ec144284a9bbfe418df4f6b61c.1629235576.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

* for-5.15/io_uring:
  io_uring: optimise io_prep_linked_timeout()
  io_uring: cancel not-armed linked touts separately
  io_uring: simplify io_prep_linked_timeout
  io_uring: kill REQ_F_LTIMEOUT_ACTIVE
  io_uring: deduplicate cancellation code
  io_uring: kill not necessary resubmit switch
  io_uring: optimise initial ltimeout refcounting
  io_uring: don't inflight-track linked timeouts
  io_uring: optimise iowq refcounting

* for-5.15/block:
  block: unexport blk_register_queue
  blk-cgroup: stop using seq_get_buf
  blk-cgroup: refactor blkcg_print_stat
  nvme: use bvec_virt
  dcssblk: use bvec_virt
  dasd: use bvec_virt
  ps3vram: use bvec_virt
  ubd: use bvec_virt
  sd: use bvec_virt
  bcache: use bvec_virt
  virtio_blk: use bvec_virt
  rbd: use bvec_virt
  squashfs: use bvec_virt
  dm-integrity: use bvec_virt
  dm-ebs: use bvec_virt
  dm: make EBS depend on !HIGHMEM
  block: use bvec_virt in bio_integrity_{process,free}
  bvec: add a bvec_virt helper
  block: ensure the bdi is freed after inode_detach_wb
  block: free the extended dev_t minor later

* for-5.15/libata:
  ata: sata_dwc_460ex: No need to call phy_exit() befre phy_init()

Linked timeout handling during issuing is heavy, it adds extra
instructions and forces to save the next linked timeout before
io_issue_sqe().

Follwing the same reasoning as in refcounting patches, a request can't
be freed by the time it returns from io_issue_sqe(), so now we don't
need to do io_prep_linked_timeout() in advance, and it can be delayed to
colder paths optimising the generic path.

Also, it should also save quite a lot for requests with linked timeouts
and completed inline on timeout spinlocking + hrtimer_start() +
hrtimer_try_to_cancel() and so on.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/19bfc9a0d26c5c5f1e359f7650afe807ca8ef879.1628981736.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Not actually used in any modular code.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210816123649.601591-1-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

seq_get_buf is a crutch that undoes all the memory safety of the
seq_file interface.  Use the normal seq_printf interfaces instead.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/20210810152623.1796144-2-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Factor out a helper to deal with a single blkcg_gq to make the code a
little bit easier to follow.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/r/20210810152623.1796144-1-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Keith Busch <kbusch@kernel.org>
Link: https://lore.kernel.org/r/20210804095634.460779-16-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-15-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Stefan Haberland <sth@linux.ibm.com>
Link: https://lore.kernel.org/r/20210804095634.460779-14-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-13-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-By: Anton Ivanov <anton.ivanov@cambridgegreys.com>
Link: https://lore.kernel.org/r/20210804095634.460779-12-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Martin K. Petersen <martin.petersen@oracle.com>
Link: https://lore.kernel.org/r/20210804095634.460779-11-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.  Note that the existing code is
fine despite ignoring bv_offset as the bio is known to contain exactly
one page from the page allocator per bio_vec.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Coly Li <colyli@suse.de>
Link: https://lore.kernel.org/r/20210804095634.460779-10-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Link: https://lore.kernel.org/r/20210804095634.460779-9-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jeff Layton <jlayton@kernel.org>
Link: https://lore.kernel.org/r/20210804095634.460779-8-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-7-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-6-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use bvec_virt instead of open coding it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-5-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

__ebs_rw_bvec use page_address on the submitted bios data, and thus
can't deal with highmem.  Disable the target on highmem configs.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20210804095634.460779-4-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Use the bvec_virt helper to clean up the bio integrity processing a
little bit.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Chaitanya Kulkarni <kch@kernel.org>
Acked-by: Martin K. Petersen <martin.petersen@oracle.com>
Link: https://lore.kernel.org/r/20210804095634.460779-3-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>