Most kmem_cache_alloc() calls are from user context. With instrumentation
enabled, the measured amount of kmem_cache_alloc() calls from non-task
context was about 0.01% of the total.

The irq disable/enable sequence used in this case to access content
from object stock is slow.  To optimize for user context access, there
are now two object stocks for task context and interrupt context access
respectively.

The task context object stock can be accessed after disabling preemption
which is cheap in non-preempt kernel. The interrupt context object stock
can only be accessed after disabling interrupt. User context code can
access interrupt object stock, but not vice versa.

The mod_objcg_state() function is also modified to make sure that memcg
and lruvec stat updates are done with interrupted disabled.

The downside of this change is that there are more data stored in local
object stocks and not reflected in the charge counter and the vmstat
arrays.  However, this is a small price to pay for better performance.

Signed-off-by: Waiman Long <longman@redhat.com>
Acked-by: Roman Gushchin <guro@fb.com>

The object stock data stored in struct memcg_stock_pcp are independent
of the other page based data stored there. Separating them out into
their own struct to highlight the independency.

Signed-off-by: Waiman Long <longman@redhat.com>
Acked-by: Roman Gushchin <guro@fb.com>

Before the new slab memory controller with per object byte charging,
charging and vmstat data update happen only when new slab pages are
allocated or freed. Now they are done with every kmem_cache_alloc()
and kmem_cache_free(). This causes additional overhead for workloads
that generate a lot of alloc and free calls.

The memcg_stock_pcp is used to cache byte charge for a specific
obj_cgroup to reduce that overhead. To further reducing it, this patch
makes the vmstat data cached in the memcg_stock_pcp structure as well
until it accumulates a page size worth of update or when other cached
data change.

On a 2-socket Cascade Lake server with instrumentation enabled and this
patch applied, it was found that about 17% (946796 out of 5515184) of the
time when __mod_obj_stock_state() is called leads to an actual call to
mod_objcg_state() after initial boot. When doing parallel kernel build,
the figure was about 16% (21894614 out of 139780628). So caching the
vmstat data reduces the number of calls to mod_objcg_state() by more
than 80%.

Signed-off-by: Waiman Long <longman@redhat.com>

In memcg_slab_free_hook()/pcpu_memcg_free_hook(), obj_cgroup_uncharge()
is followed by mod_objcg_state()/mod_memcg_state(). Each of these
function call goes through a separate irq_save/irq_restore cycle. That
is inefficient.  Introduce a new function obj_cgroup_uncharge_mod_state()
that combines them with a single irq_save/irq_restore cycle.

Signed-off-by: Waiman Long <longman@redhat.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Acked-by: Roman Gushchin <guro@fb.com>

The caller of mod_memcg_lruvec_state() has both memcg and lruvec readily
available. So both of them are now passed to mod_memcg_lruvec_state()
and __mod_memcg_lruvec_state(). The __mod_memcg_lruvec_state() is
updated to allow either of the two parameters to be set to null. This
makes mod_memcg_lruvec_state() equivalent to mod_memcg_state() if lruvec
is null.

The new __mod_memcg_lruvec_state() function will be used in the next
patch as a replacement of mod_memcg_state() in mm/percpu.c for the
consolidation of the memory uncharge and vmstat update functions in
the kmem_cache_free() path.

Signed-off-by: Waiman Long <longman@redhat.com>
Acked-by: Roman Gushchin <guro@fb.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>

nr_empty_pop_pages is used to guarantee that there are some free
populated pages to satisfy atomic allocations. Accounted and
non-accounted allocations are using separate sets of chunks,
so both need to have a surplus of empty pages.

This commit makes pcpu_nr_empty_pop_pages and the corresponding logic
per chunk type.

[Dennis]
This issue came up as I was reviewing [1] and realized I missed this.
Simultaneously, it was reported btrfs was seeing failed atomic
allocations in fsstress tests [2] and [3].

[1] https://lore.kernel.org/linux-mm/20210324190626.564297-1-guro@fb.com/
[2] https://lore.kernel.org/linux-mm/20210401185158.3275.409509F4@e16-tech.com/
[3] https://lore.kernel.org/linux-mm/CAL3q7H5RNBjCi708GH7jnczAOe0BLnacT9C+OBgA-Dx9jhB6SQ@mail.gmail.com/

Fixes: 3c7be18 ("mm: memcg/percpu: account percpu memory to memory cgroups")
Cc: stable@vger.kernel.org # 5.9+
Signed-off-by: Roman Gushchin <guro@fb.com>
Tested-by: Filipe Manana <fdmanana@suse.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>

For each device, the nosy driver allocates a pcilynx structure.
A use-after-free might happen in the following scenario:

 1. Open nosy device for the first time and call ioctl with command
    NOSY_IOC_START, then a new client A will be malloced and added to
    doubly linked list.
 2. Open nosy device for the second time and call ioctl with command
    NOSY_IOC_START, then a new client B will be malloced and added to
    doubly linked list.
 3. Call ioctl with command NOSY_IOC_START for client A, then client A
    will be readded to the doubly linked list. Now the doubly linked
    list is messed up.
 4. Close the first nosy device and nosy_release will be called. In
    nosy_release, client A will be unlinked and freed.
 5. Close the second nosy device, and client A will be referenced,
    resulting in UAF.

The root cause of this bug is that the element in the doubly linked list
is reentered into the list.

Fix this bug by adding a check before inserting a client.  If a client
is already in the linked list, don't insert it.

The following KASAN report reveals it:

   BUG: KASAN: use-after-free in nosy_release+0x1ea/0x210
   Write of size 8 at addr ffff888102ad7360 by task poc
   CPU: 3 PID: 337 Comm: poc Not tainted 5.12.0-rc5+ torvalds#6
   Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
   Call Trace:
     nosy_release+0x1ea/0x210
     __fput+0x1e2/0x840
     task_work_run+0xe8/0x180
     exit_to_user_mode_prepare+0x114/0x120
     syscall_exit_to_user_mode+0x1d/0x40
     entry_SYSCALL_64_after_hwframe+0x44/0xae

   Allocated by task 337:
     nosy_open+0x154/0x4d0
     misc_open+0x2ec/0x410
     chrdev_open+0x20d/0x5a0
     do_dentry_open+0x40f/0xe80
     path_openat+0x1cf9/0x37b0
     do_filp_open+0x16d/0x390
     do_sys_openat2+0x11d/0x360
     __x64_sys_open+0xfd/0x1a0
     do_syscall_64+0x33/0x40
     entry_SYSCALL_64_after_hwframe+0x44/0xae

   Freed by task 337:
     kfree+0x8f/0x210
     nosy_release+0x158/0x210
     __fput+0x1e2/0x840
     task_work_run+0xe8/0x180
     exit_to_user_mode_prepare+0x114/0x120
     syscall_exit_to_user_mode+0x1d/0x40
     entry_SYSCALL_64_after_hwframe+0x44/0xae

   The buggy address belongs to the object at ffff888102ad7300 which belongs to the cache kmalloc-128 of size 128
   The buggy address is located 96 bytes inside of 128-byte region [ffff888102ad7300, ffff888102ad7380)

[ Modified to use 'list_empty()' inside proper lock  - Linus ]

Link: https://lore.kernel.org/lkml/1617433116-5930-1-git-send-email-zheyuma97@gmail.com/
Reported-and-tested-by: 马哲宇 (Zheyu Ma) <zheyuma97@gmail.com>
Signed-off-by: Zheyu Ma <zheyuma97@gmail.com>
Cc: Greg Kroah-Hartman <greg@kroah.com>
Cc: Stefan Richter <stefanr@s5r6.in-berlin.de>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Pull OpenRISC fix from Stafford Horne:
 "Fix duplicate header include in Litex SOC driver"

* tag 'for-linus' of git://github.com/openrisc/linux:
  soc: litex: Remove duplicated header file inclusion

POull io_uring fix from Jens Axboe:
 "Just fixing a silly braino in a previous patch, where we'd end up
  failing to compile if CONFIG_BLOCK isn't enabled.

  Not that a lot of people do that, but kernel bot spotted it and it's
  probably prudent to just flush this out now before -rc6.

  Sorry about that, none of my test compile configs have !CONFIG_BLOCK"

* tag 'io_uring-5.12-2021-04-03' of git://git.kernel.dk/linux-block:
  io_uring: fix !CONFIG_BLOCK compilation failure

The header file <linux/errno.h> is already included above and can be
removed here.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
Signed-off-by: Mateusz Holenko <mholenko@antmicro.com>
Signed-off-by: Stafford Horne <shorne@gmail.com>

…ux/kernel/git/gfs2/linux-gfs2

Pull gfs2 fixes from Andreas Gruenbacher:
 "Two more gfs2 fixes"

* tag 'gfs2-v5.12-rc2-fixes2' of git://git.kernel.org/pub/scm/linux/kernel/git/gfs2/linux-gfs2:
  gfs2: report "already frozen/thawed" errors
  gfs2: Flag a withdraw if init_threads() fails

…linux/kernel/git/riscv/linux

Pull RISC-V fixes from Palmer Dabbelt:
 "A handful of fixes for 5.12:

   - fix a stack tracing regression related to "const register asm"
     variables, which have unexpected behavior.

   - ensure the value to be written by put_user() is evaluated before
     enabling access to userspace memory..

   - align the exception vector table correctly, so we don't rely on the
     firmware's handling of unaligned accesses.

   - build fix to make NUMA depend on MMU, which triggered on some
     randconfigs"

* tag 'riscv-for-linus-5.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux:
  riscv: Make NUMA depend on MMU
  riscv: remove unneeded semicolon
  riscv,entry: fix misaligned base for excp_vect_table
  riscv: evaluate put_user() arg before enabling user access
  riscv: Drop const annotation for sp

…el/git/powerpc/linux

Pull powerpc fixes from Michael Ellerman:
 "Fix a bug on pseries where spurious wakeups from H_PROD would prevent
  partition migration from succeeding.

  Fix oopses seen in pcpu_alloc(), caused by parallel faults of the
  percpu mapping causing us to corrupt the protection key used for the
  mapping, and cause a fatal key fault.

  Thanks to Aneesh Kumar K.V, Murilo Opsfelder Araujo, and Nathan Lynch"

* tag 'powerpc-5.12-5' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux:
  powerpc/mm/book3s64: Use the correct storage key value when calling H_PROTECT
  powerpc/pseries/mobility: handle premature return from H_JOIN
  powerpc/pseries/mobility: use struct for shared state

…scm/linux/kernel/git/hyperv/linux

Pull Hyper-V fixes from Wei Liu:
 "One fix from Lu Yunlong for a double free in hvfb_probe"

* tag 'hyperv-fixes-signed-20210402' of git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux:
  video: hyperv_fb: Fix a double free in hvfb_probe

…x/kernel/git/gregkh/driver-core

Pull driver core fix from Greg KH:
 "Here is a single driver core fix for a reported problem with differed
  probing. It has been in linux-next for a while with no reported
  problems"

* tag 'driver-core-5.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core:
  driver core: clear deferred probe reason on probe retry

…kernel/git/gregkh/char-misc

Pull char/misc driver fixes from Greg KH:
 "Here are a few small driver char/misc changes for 5.12-rc6.

  Nothing major here, a few fixes for reported issues:

   - interconnect fixes for problems found

   - fbcon syzbot-found fix

   - extcon fixes

   - firmware stratix10 bugfix

   - MAINTAINERS file update.

  All of these have been in linux-next for a while with no reported
  issues"

* tag 'char-misc-5.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc:
  drivers: video: fbcon: fix NULL dereference in fbcon_cursor()
  mei: allow map and unmap of client dma buffer only for disconnected client
  MAINTAINERS: Add linux-phy list and patchwork
  interconnect: Fix kerneldoc warning
  firmware: stratix10-svc: reset COMMAND_RECONFIG_FLAG_PARTIAL to 0
  extcon: Fix error handling in extcon_dev_register
  extcon: Add stubs for extcon_register_notifier_all() functions
  interconnect: core: fix error return code of icc_link_destroy()
  interconnect: qcom: msm8939: remove rpm-ids from non-RPM nodes

…rnel/git/gregkh/staging

Pull staging driver fixes from Greg KH:
 "Here are two rtl8192e staging driver fixes for reported problems.

  Both of these have been in linux-next for a while with no reported
  issues"

* tag 'staging-5.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging:
  staging: rtl8192e: Change state information from u16 to u8
  staging: rtl8192e: Fix incorrect source in memcpy()

…/git/gregkh/tty

Pull serial driver fix from Greg KH:
 "Here is a single serial driver fix for 5.12-rc6. Is is a revert of a
  change that showed up in 5.9 that has been reported to cause problems.

  It has been in linux-next for a while with no reported issues"

* tag 'tty-5.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty:
  soc: qcom-geni-se: Cleanup the code to remove proxy votes

…/git/gregkh/usb

Pull USB fixes from Greg KH:
 "Here are a few small USB driver fixes for 5.12-rc6 to resolve reported
  problems.

  They include:

   - a number of cdc-acm fixes for reported problems. It seems more
     people are using this driver lately...

   - dwc3 driver fixes for reported problems, and fixes for the fixes :)

   - dwc2 driver fixes for reported issues.

   - musb driver fix.

   - new USB quirk additions.

  All of these have been in linux-next for a while with no reported
  issues"

* tag 'usb-5.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb: (23 commits)
  usb: dwc2: Prevent core suspend when port connection flag is 0
  usb: dwc2: Fix HPRT0.PrtSusp bit setting for HiKey 960 board.
  usb: musb: Fix suspend with devices connected for a64
  usb: xhci-mtk: fix broken streams issue on 0.96 xHCI
  usb: dwc3: gadget: Clear DEP flags after stop transfers in ep disable
  usbip: vhci_hcd fix shift out-of-bounds in vhci_hub_control()
  USB: quirks: ignore remote wake-up on Fibocom L850-GL LTE modem
  USB: cdc-acm: do not log successful probe on later errors
  USB: cdc-acm: always claim data interface
  USB: cdc-acm: use negation for NULL checks
  USB: cdc-acm: clean up probe error labels
  USB: cdc-acm: drop redundant driver-data reset
  USB: cdc-acm: drop redundant driver-data assignment
  USB: cdc-acm: fix use-after-free after probe failure
  USB: cdc-acm: fix double free on probe failure
  USB: cdc-acm: downgrade message to debug
  USB: cdc-acm: untangle a circular dependency between callback and softint
  cdc-acm: fix BREAK rx code path adding necessary calls
  usb: gadget: udc: amd5536udc_pci fix null-ptr-dereference
  usb: dwc3: pci: Enable dis_uX_susphy_quirk for Intel Merrifield
  ...

…it/jejb/scsi

Pull SCSI fix from James Bottomley:
 "A single fix to iscsi for a rare race condition which can cause a
  kernel panic"

* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
  scsi: iscsi: Fix race condition between login and sync thread

kernel test robot correctly pinpoints a compilation failure if
CONFIG_BLOCK isn't set:

fs/io_uring.c: In function '__io_complete_rw':
>> fs/io_uring.c:2509:48: error: implicit declaration of function 'io_rw_should_reissue'; did you mean 'io_rw_reissue'? [-Werror=implicit-function-declaration]
    2509 |  if ((res == -EAGAIN || res == -EOPNOTSUPP) && io_rw_should_reissue(req)) {
         |                                                ^~~~~~~~~~~~~~~~~~~~
         |                                                io_rw_reissue
    cc1: some warnings being treated as errors

Ensure that we have a stub declaration of io_rw_should_reissue() for
!CONFIG_BLOCK.

Fixes: 230d50d ("io_uring: move reissue into regular IO path")
Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

Pull block fixes from Jens Axboe:

 - Remove comment that never came to fruition in 22 years of development
   (Christoph)

 - Remove unused request flag (Christoph)

 - Fix for null_blk fake timeout handling (Damien)

 - Fix for IOCB_NOWAIT being ignored for O_DIRECT on raw bdevs (Pavel)

 - Error propagation fix for multiple split bios (Yufen)

* tag 'block-5.12-2021-04-02' of git://git.kernel.dk/linux-block:
  block: remove the unused RQF_ALLOCED flag
  block: update a few comments in uapi/linux/blkpg.h
  block: don't ignore REQ_NOWAIT for direct IO
  null_blk: fix command timeout completion handling
  block: only update parent bi_status when bio fail

Pull io_uring fixes from Jens Axboe:
 "Nothing really major in here, and finally nothing really related to
  signals. A few minor fixups related to the threading changes, and some
  general fixes, that's it.

  There's the pending gdb-get-confused-about-arch, but that's more of a
  cosmetic issue, nothing that hinder use of it. And given that other
  archs will likely be affected by that oddity too, better to postpone
  any changes there until 5.13 imho"

* tag 'io_uring-5.12-2021-04-02' of git://git.kernel.dk/linux-block:
  io_uring: move reissue into regular IO path
  io_uring: fix EIOCBQUEUED iter revert
  io_uring/io-wq: protect against sprintf overflow
  io_uring: don't mark S_ISBLK async work as unbounded
  io_uring: drop sqd lock before handling signals for SQPOLL
  io_uring: handle setup-failed ctx in kill_timeouts
  io_uring: always go for cancellation spin on exec

…l/git/rafael/linux-pm

Pull ACPI fixes from Rafael Wysocki:
 "These fix an ACPI tables management issue, an issue related to the
  ACPI enumeration of devices and CPU wakeup in the ACPI processor
  driver.

  Specifics:

   - Ensure that the memory occupied by ACPI tables on x86 will always
     be reserved to prevent it from being allocated for other purposes
     which was possible in some cases (Rafael Wysocki).

   - Fix the ACPI device enumeration code to prevent it from attempting
     to evaluate the _STA control method for devices with unmet
     dependencies which is likely to fail (Hans de Goede).

   - Fix the handling of CPU0 wakeup in the ACPI processor driver to
     prevent CPU0 online failures from occurring (Vitaly Kuznetsov)"

* tag 'acpi-5.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
  ACPI: processor: Fix CPU0 wakeup in acpi_idle_play_dead()
  ACPI: scan: Fix _STA getting called on devices with unmet dependencies
  ACPI: tables: x86: Reserve memory occupied by ACPI tables

…git/rafael/linux-pm

Pull power management fixes from Rafael Wysocki:
 "These fix a race condition and an ordering issue related to using
  device links in the runtime PM framework and two kerneldoc comments in
  cpufreq.

  Specifics:

   - Fix race condition related to the handling of supplier devices
     during consumer device probe and fix the order of decrementation of
     two related reference counters in the runtime PM core code handling
     supplier devices (Adrian Hunter).

   - Fix kerneldoc comments in cpufreq that have not been updated along
     with the functions documented by them (Geert Uytterhoeven)"

* tag 'pm-5.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm:
  PM: runtime: Fix race getting/putting suppliers at probe
  PM: runtime: Fix ordering in pm_runtime_get_suppliers()
  cpufreq: Fix scaling_{available,boost}_frequencies_show() comments

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

The big top of the file comment talk about grand plans that never
happened, so remove them to not confuse the readers.  Also mark the
devname and volname fields as ignored as they were never used by the
kernel.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

…ernel/git/rostedt/linux-trace

Pull tracing fix from Steven Rostedt:
 "Fix stack trace entry size to stop showing garbage

  The macro that creates both the structure and the format displayed to
  user space for the stack trace event was changed a while ago to fix
  the parsing by user space tooling. But this change also modified the
  structure used to store the stack trace event. It changed the caller
  array field from [0] to [8].

  Even though the size in the ring buffer is dynamic and can be
  something other than 8 (user space knows how to handle this), the 8
  extra words was not accounted for when reserving the event on the ring
  buffer, and added 8 more entries, due to the calculation of
  "sizeof(*entry) + nr_entries * sizeof(long)", as the sizeof(*entry)
  now contains 8 entries.

  The size of the caller field needs to be subtracted from the size of
  the entry to create the correct allocation size"

* tag 'trace-v5.12-rc5-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace:
  tracing: Fix stack trace event size

It's non-obvious how retry is done for block backed files, when it happens
off the kiocb done path. It also makes it tricky to deal with the iov_iter
handling.

Just mark the req as needing a reissue, and handling it from the
submission path instead. This makes it directly obvious that we're not
re-importing the iovec from userspace past the submit point, and it means
that we can just reuse our usual -EAGAIN retry path from the read/write
handling.

At some point in the future, we'll gain the ability to always reliably
return -EAGAIN through the stack. A previous attempt on the block side
didn't pan out and got reverted, hence the need to check for this
information out-of-band right now.

Signed-off-by: Jens Axboe <axboe@kernel.dk>

* acpi-tables:
  ACPI: tables: x86: Reserve memory occupied by ACPI tables

* acpi-scan:
  ACPI: scan: Fix _STA getting called on devices with unmet dependencies

* pm-cpufreq:
  cpufreq: Fix scaling_{available,boost}_frequencies_show() comments

If IOCB_NOWAIT is set on submission, then that needs to get propagated to
REQ_NOWAIT on the block side. Otherwise we completely lose this
information, and any issuer of IOCB_NOWAIT IO will potentially end up
blocking on eg request allocation on the storage side.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

NUMA is useless when NOMMU, and it leads some build error,
make it depend on MMU.

Reported-by: kernel test robot <lkp@intel.com>
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>