Implementaion is based on existing read and write infrastructure.
copy_max_bytes: A new configfs and module parameter is introduced, which
can be used to set hardware/driver supported maximum copy limit.

Suggested-by: Damien Le Moal <damien.lemoal@opensource.wdc.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>

Setting copy_offload_supported flag to enable offload.

Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>

Before enabling copy for dm target, check if underlying devices and
dm target support copy. Avoid split happening inside dm target.
Fail early if the request needs split, currently splitting copy
request is not supported.

Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>

Add support for handling target command on target.
For bdev-ns we call into blkdev_issue_copy, which the block layer
completes by a offloaded copy request to backend bdev or by emulating the
request.

For file-ns we call vfs_copy_file_range to service our request.

Currently target always shows copy capability by setting
NVME_CTRL_ONCS_COPY in controller ONCS.

Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>

For device supporting native copy, nvme driver receives read and
write request with BLK_COPY op flags.
For read request the nvme driver populates the payload with source
information.
For write request the driver converts it to nvme copy command using the
source information in the payload and submits to the device.
current design only supports single source range.
This design is courtesy Mikulas Patocka's token based copy

trace event support for nvme_copy_cmd.
Set the device copy limits to queue limits.

Signed-off-by: Kanchan Joshi <joshi.k@samsung.com>
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Javier González <javier.gonz@samsung.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>

For direct block device opened with O_DIRECT, use copy_file_range to
issue device copy offload, and fallback to generic_copy_file_range incase
device copy offload capability is absent.
Modify checks to allow bdevs to use copy_file_range.

Suggested-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>

For the devices which does not support copy, copy emulation is added.
It is required for in-kernel users like fabrics, where file descriptor is
not available and hence they can't use copy_file_range.
Copy-emulation is implemented by reading from source into memory and
writing to the corresponding destination asynchronously.
Also emulation is used, if copy offload fails or partially completes.

Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Vincent Fu <vincent.fu@samsung.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>

Introduce blkdev_issue_copy which takes similar arguments as
copy_file_range and performs copy offload between two bdevs.
Introduce REQ_COPY copy offload operation flag. Create a read-write
bio pair with a token as payload and submitted to the device in order.
Read request populates token with source specific information which
is then passed with write request.
This design is courtesy Mikulas Patocka's token based copy

Larger copy will be divided, based on max_copy_sectors limit.

Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>

Add device limits as sysfs entries,
        - copy_offload (RW)
        - copy_max_bytes (RW)
        - copy_max_bytes_hw (RO)

Above limits help to split the copy payload in block layer.
copy_offload: used for setting copy offload(1) or emulation(0).
copy_max_bytes: maximum total length of copy in single payload.
copy_max_bytes_hw: Reflects the device supported maximum limit.

Reviewed-by: Hannes Reinecke <hare@suse.de>
Signed-off-by: Nitesh Shetty <nj.shetty@samsung.com>
Signed-off-by: Kanchan Joshi <joshi.k@samsung.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>

Signed-off-by: Mark Brown <broonie@kernel.org>

Commit 9fd4968 ("bpftool: Support inline annotations when dumping the CFG of a program")
breaks the build of the perf tool on s390 in the linux-next repository.
Here is the make output:

make -C tools/perf
....
btf_dumper.c: In function 'dotlabel_puts':
DEBUG: btf_dumper.c:838:25: error: '__fallthrough' undeclared \
		(first use in this function); did you mean 'fallthrough'?
DEBUG:   838 |                         __fallthrough;
DEBUG:       |                         ^~~~~~~~~~~~~
DEBUG:       |                         fallthrough
DEBUG: btf_dumper.c:838:25: note: each undeclared identifier is reported \
		only once for each function it appears in
DEBUG: btf_dumper.c:837:25: warning: this statement may fall through \
                [-Wimplicit-fallthrough=]
DEBUG:   837 |                         putchar('\\');
DEBUG:       |                         ^~~~~~~~~~~~~
DEBUG: btf_dumper.c:839:17: note: here
DEBUG:   839 |                 default:
DEBUG:       |                 ^~~~~~~
DEBUG: make[3]: *** [Makefile:247: /builddir/build/BUILD/kernel-6.2.fc37/\
		        linux-6.2/tools/perf/util/bpf_skel/ \
		        .tmp/bootstrap/btf_dumper.o] Error 1

The compile fails because symbol __fallthrough unknown, but symbol
fallthrough is known and works fine.

Fix this and replace __fallthrough by fallthrough.

With this change, the compile works.

Output after:

 # make -C tools/perf
 ....
 CC      util/bpf-filter.o
 CC      util/bpf-filter-flex.o
 LD      util/perf-in.o
 LD      perf-in.o
 LINK    perf
 make: Leaving directory '/root/mirror-linux-next/tools/perf'
 #

Fixes: 9fd4968 ("bpftool: Support inline annotations when dumping the CFG of a program")
Signed-off-by: Thomas Richter <tmricht@linux.ibm.com>
Acked-by: Quentin Monnet <quentin@isovalent.com>
Signed-off-by: Mark Brown <broonie@kernel.org>

…git/jgg/iommufd.git

…rnel/git/gustavoars/linux.git

# Conflicts:
#	include/linux/cpumask.h

…el/git/mcgrof/linux.git

…mic/linux.git

…git/vbabka/slab.git

# Conflicts:
#	include/linux/slab.h
#	mm/slob.c

…git/krisman/unicode.git

…efi/efi.git

…git/rppt/memblock.git

…git/mani/mhi.git

…git/srini/nvmem.git

# Conflicts:
#	drivers/nvmem/layouts/sl28vpd.c
#	drivers/of/Makefile
#	include/linux/of.h

…git/abelloni/linux.git

…git/livepatching/livepatching

…/shuah/linux-kselftest.git

…shuah/linux-kselftest.git

…git/ebiederm/user-namespace.git

…git/thierry.reding/linux-pwm.git

…git/linusw/linux-pinctrl.git

…rnel/git/brgl/linux.git

…git/remoteproc/linux.git