Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ppc64_defconfig zImage fails to boot in qemu #402

Open
shenki opened this issue Mar 30, 2022 · 8 comments
Open

ppc64_defconfig zImage fails to boot in qemu #402

shenki opened this issue Mar 30, 2022 · 8 comments
Labels
bug It's a bug enhancement Addition/modification of a feature, not a bug per se in-progress Some work has been done on this issue

Comments

@shenki
Copy link
Member

shenki commented Mar 30, 2022

v5.18-rc0 ppc64_defconfig fails to boot in qemu.

built with REHL9:

  • powerpc64le-linux-gnu-gcc (GCC) 11.2.1 20210728 (Red Hat Cross 11.2.1-1)
  • GNU ld version 2.35.2-1.fc34
qemu-system-ppc64 -nographic -nodefaults -M pseries,x-vof=off -kernel arch/powerpc/boot/zImage.pseries -serial mon:stdio 
( 700 ) Program Exception [ 1daf04c4 ]


    R0 .. R7           R8 .. R15         R16 .. R23         R24 .. R31
8000000000002000   000000001db73400   0000000000000000   0000000000000000   
000000001db7c400   8000000000000000   0000000000000000   0000000000000000   
0000000000000000   000000001db6e400   0000000000000000   0000000000000000   
0000000000000000   000000001daf449c   0000000000000000   0000000000000000   
0000000000000000   0000000040000008   0000000000000000   0000000000000000   
000000001daf04c4   0000000000000000   0000000000000000   0000000000000000   
0000000000000000   0000000000000000   0000000000000000   0000000000000000   
0000000000000000   0000000000000000   0000000000000000   0000000000000000   

    CR / XER           LR / CTR          SRR0 / SRR1        DAR / DSISR
        80000000   000000001daf04f0   0000000000400000   0000000000000000   
0000000020040000   0000000000400000   8000000000082000           00000000   

emu-system-ppc64 -nographic -nodefaults -M powernv9 -kernel arch/powerpc/boot/zImage.epapr -serial mon:stdio
   10.409830632,5] INIT: Starting kernel at 0x20010000, fdt at 0x306a6120 27842 bytes
[   10.481490807,3] ***********************************************
[   11.009953442,3] Fatal Exception 0xe40 at 000000003018af00  __toc_start+0x0  MSR 1000000000000000
[   11.010243386,3] CFAR : 000000002001c890 MSR  : 1000000000000000
[   11.010427333,3] SRR0 : 0000000030020214 SRR1 : 9000000000191002
[   11.010602706,3] HSRR0: 000000003018af00 HSRR1: 1000000000000000
[   11.010775543,3] DSISR: 00000000         DAR  : 0000000000000000
[   11.010947568,3] LR   : 0000000000000016 CTR  : 000000002001baa0
[   11.011150324,3] CR   : 42002804         XER  : 20040000
[   11.011311045,3] GPR00: 0000000000000019 GPR16: 0000000031c10000
[   11.011504043,3] GPR01: 00000000209f5bdc GPR17: 00000000300031bc
[   11.011681148,3] GPR02: 00000000300215d4 GPR18: 0000000000000000
[   11.011842899,3] GPR03: 0000000000000000 GPR19: 0000000000000000
[   11.012017027,3] GPR04: 0000000000000000 GPR20: 0000000000000000
[   11.012190884,3] GPR05: 00000000209f5ef4 GPR21: 0000000000000000
[   11.012355844,3] GPR06: 00000000209f60e1 GPR22: 000000003013f863
[   11.012536238,3] GPR07: 00000000209f5e74 GPR23: 0000000030120ece
[   11.012715833,3] GPR08: 0000000000000008 GPR24: 00000000301205f5
[   11.012879361,3] GPR09: 000000002001baa0 GPR25: 00000000209f6064
[   11.013054633,3] GPR10: 0000000000000008 GPR26: 00000000209f6028
[   11.013228830,3] GPR11: 0000001600000000 GPR27: 00000000209f65c8
[   11.013405767,3] GPR12: 000000003018af00 GPR28: 00000000209f5ef4
[   11.013582507,3] GPR13: 000000003018af00 GPR29: 00000000209f6084
[   11.013757789,3] GPR14: 0000000000000000 GPR30: 0000000020026e60
[   11.013919463,3] GPR15: 0000000000000000 GPR31: 00000000209f60ac
CPU 0000 Backtrace:
 S: 00000000209f5eec R: 000000002001bafc ! 
kernel pseries powernv
v5.18-rc0 (ae085d7f9365) x x
v5.17 x x
v5.4 x
v4.19 x
v4.14 x

Just in case, I checked with the powerpc64-linux-gnu-gcc to make sure it wasn't the le variant of the cross compiler causing an issue.

@chleroy
Copy link

chleroy commented Mar 30, 2022

I guess these days nobody is booting zImage in QEMU, we are using vmlinux

See https://github.com/linuxppc/wiki/wiki/Booting-with-Qemu

@chleroy chleroy changed the title ppc64_defconfig fails to boot in qemu ppc64_defconfig zImage fails to boot in qemu Mar 30, 2022
@shenki
Copy link
Member Author

shenki commented Mar 30, 2022

I guess these days nobody is booting zImage in QEMU, we are using vmlinux

I was trying to test a change to the wrapper. If it does not work then perhaps we should disable it for big endian?

@mpe
Copy link
Member

mpe commented Mar 30, 2022

The zImage built for ppc64_defconfig is a 32-bit binary. It seems that SLOF/qemu isn't loading it properly.

If I drop into SLOF and dump the ppc64le image, I see:

SLOF **********************************************************************
QEMU Starting
...
Ready!
0 > 400000 10 ldump
00400000: 48000008 2c000048 7d6000a6 696b0001 ok

Which matches the zImage text:

Disassembly of section .text:

0000000000000000 <_start>:
       0:       48 00 00 08     tdi     0,r0,72
       4:       2c 00 00 48     b       30 <RELA+0x29>

In contrast when booting the big endian zImage, I see:

SLOF **********************************************************************
QEMU Starting
...
Ready!
Ready!
0 > 400000 10 ldump
00400000: 00000000 00000000 00000000 00000000 ok

So when we jump to that it's all zeroes and we take a illegal instruction program check.

I'm not sure whether that's a qemu or SLOF issue, cc @aik

@shenki
Copy link
Member Author

shenki commented May 5, 2022

@aik appears to be working on a qemu fix here: https://lore.kernel.org/qemu-devel/20220504065536.3534488-1-aik@ozlabs.ru/

His patch fixes the pseries qemu machine when running with x-vof=on.

In the commit message he mentions what works and what doesn't:

Note #1: SLOF (x-vof=off) still cannot boot a big endian zImage as
SLOF enables MSR_SF for everything loaded by QEMU and this leads to early
crash of 32bit zImage.

Note #2: BE/LE vmlinux images set MSR_SF in early boot so these just work;
a LE zImage restores MSR_SF after every CI call and we are lucky enough
not to crash before the first CI call.

@mpe
Copy link
Member

mpe commented May 5, 2022

One thing we could try is making PPC64_BOOT_WRAPPER user selectable, and turn that on for BE builds. That would mean the BE zImage could be 64-bit, like the LE one.

@mpe mpe added enhancement Addition/modification of a feature, not a bug per se in-progress Some work has been done on this issue bug It's a bug labels May 5, 2022
@shenki
Copy link
Member Author

shenki commented May 5, 2022

kernel pseries,x-vof=off pseries,x-vof=on powernv
vmlinux BE
vmlinux LE
zImage.pseries BE x ✓¹ N/A
zImage.pseries LE ✓²
zImage.epapr BE N/A N/A x
zImage.epapr LE N/A N/A

Notes:
¹ works only with Alexey's qemu patch applied, and must set kernel-addr=0
² the pseries wrapper includes the epapr wrapper, and will detect epapr boot successfully

Tested with qemu 7.0 and Linux v5.18-rc5.

@aik
Copy link
Member

aik commented May 5, 2022

The zImage built for ppc64_defconfig is a 32-bit binary. It seems that SLOF/qemu isn't loading it properly.

btw QEMU has trace points (enable via -trace events=/home/aik/qemu_trace_events) and with the defaults, loader_write_rom gives a hint of what goes wrong:

loader_write_rom slof.bin: @0x0 size=0xe3350 ROM=0                                        
loader_write_rom /home/aik/pbuild/kernel-be-defconfig/arch/powerpc/boot/zImage.pseries \
ELF program header segment 0: @0x4400000 size=0x9ece52 ROM=0                                  

0x4400000 is where zImage was loaded.

@shenki
Copy link
Member Author

shenki commented May 5, 2022

One thing we could try is making PPC64_BOOT_WRAPPER user selectable, and turn that on for BE builds. That would mean the BE zImage could be 64-bit, like the LE one.

I tried this and it made the linker segfault:

binutils:
  Installed: 2.38-4
$ powerpc64le-linux-gnu-ld -m elf32ppc -T arch/powerpc/boot/zImage.lds -Ttext 0x20000000 -pie --no-dynamic-linker -z notext -o arch/powerpc/boot/zImage.epapr -Map wrapper.map arch/powerpc/boot/pseries-head.o arch/powerpc/boot/epapr.o arch/powerpc/boot/epapr-wrapper.o ./zImage.2535927.o arch/powerpc/boot/wrapper.a
#0  bfd_link_hash_traverse (htab=0xaaaaba069a60, func=func@entry=0xaaaab9ec8760 <lang_one_common>, info=info@entry=0x0)
    at ../../bfd/linker.c:669
669	../../bfd/linker.c: No such file or directory.
(gdb) bt
#0  bfd_link_hash_traverse (htab=0xaaaaba069a60, func=func@entry=0xaaaab9ec8760 <lang_one_common>, info=info@entry=0x0)
    at ../../bfd/linker.c:669
#1  0x0000aaaab9ed3d84 in lang_common () at ../../ld/ldlang.c:7183
#2  lang_process () at ../../ld/ldlang.c:8193
#3  0x0000aaaab9ebe290 in main (argc=<optimized out>, argv=<optimized out>) at ../../ld/ldmain.c:497
(gdb) bt full
#0  bfd_link_hash_traverse (htab=0xaaaaba069a60, func=func@entry=0xaaaab9ec8760 <lang_one_common>, info=info@entry=0x0)
    at ../../bfd/linker.c:669
        p = 0x20
        i = 100
#1  0x0000aaaab9ed3d84 in lang_common () at ../../ld/ldlang.c:7183
No locals.
#2  lang_process () at ../../ld/ldlang.c:8193
No locals.
#3  0x0000aaaab9ebe290 in main (argc=<optimized out>, argv=<optimized out>) at ../../ld/ldmain.c:497
        emulation = 0xfffff8d005ec "elf32ppc"
        start_time = 919

The reason I went down this path was I assumed booting the wrapper in qemu would work, but it never has. That's annoying if you're testing changes in the wrapper, but it's not really important for any real machine.

It might be just as useful to document the configurations that are expected to work, so we can point users at that.

intel-lab-lkp pushed a commit to intel-lab-lkp/linux that referenced this issue Aug 8, 2023
Kernel uses ARCH=powerpc for both 32-bit and 64-bit PowerPC, here adds a
ppc64 variant for big endian 64-bit PowerPC, users can pass XARCH=ppc64
to test it.

The powernv machine of qemu-system-ppc64 is used with
powernv_be_defconfig.

As the document [1] shows:

  PowerNV (as Non-Virtualized) is the “bare metal” platform using the
  OPAL firmware. It runs Linux on IBM and OpenPOWER systems and it can be
  used as an hypervisor OS, running KVM guests, or simply as a host OS.

Notes,

- differs from little endian 64-bit PowerPC, vmlinux is used instead of
  zImage, because big endian zImage [2] only boot on qemu with x-vof=on
  (added from qemu v7.0) and a fixup patch [3] for qemu v7.0.51:

- since the VSX support may be disabled in kernel side, to avoid
  "illegal instruction" errors due to missing VSX kernel support, let's
  simply let compiler not generate vector/scalar (VSX) instructions via
  the '-mno-vsx' option.

- as 'man gcc' shows, '-mmultiple' is used to generate code that uses
  the load multiple word instructions and the store multiple word
  instructions. Those instructions do not work when the processor is in
  little-endian mode (except PPC740/PPC750), so, we only enable it
  for big endian powerpc.

- for big endian ppc64, as the help message from arch/powerpc/Kconfig
  shows, the V2 ABI is standard for 64-bit little-endian, but for
  big-endian it is less well tested by kernel and toolchain, so, use
  elfv1 as-is, no need to explicitly ask toolchain to use elfv2 here.

[1]: https://qemu.readthedocs.io/en/latest/system/ppc/powernv.html
[2]: linuxppc/issues#402
[3]: https://lore.kernel.org/qemu-devel/20220504065536.3534488-1-aik@ozlabs.ru/

Suggested-by: Willy Tarreau <w@1wt.eu>
Link: https://lore.kernel.org/lkml/20230722121019.GD17311@1wt.eu/
Link: https://lore.kernel.org/lkml/20230719043353.GC5331@1wt.eu/
Reviewed-by: Thomas Weißschuh <linux@weissschuh.net>
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
Signed-off-by: Willy Tarreau <w@1wt.eu>
roxell pushed a commit to roxell/linux that referenced this issue Aug 17, 2023
Kernel uses ARCH=powerpc for both 32-bit and 64-bit PowerPC, here adds a
ppc64 variant for big endian 64-bit PowerPC, users can pass XARCH=ppc64
to test it.

The powernv machine of qemu-system-ppc64 is used with
powernv_be_defconfig.

As the document [1] shows:

  PowerNV (as Non-Virtualized) is the “bare metal” platform using the
  OPAL firmware. It runs Linux on IBM and OpenPOWER systems and it can be
  used as an hypervisor OS, running KVM guests, or simply as a host OS.

Notes,

- differs from little endian 64-bit PowerPC, vmlinux is used instead of
  zImage, because big endian zImage [2] only boot on qemu with x-vof=on
  (added from qemu v7.0) and a fixup patch [3] for qemu v7.0.51:

- since the VSX support may be disabled in kernel side, to avoid
  "illegal instruction" errors due to missing VSX kernel support, let's
  simply let compiler not generate vector/scalar (VSX) instructions via
  the '-mno-vsx' option.

- as 'man gcc' shows, '-mmultiple' is used to generate code that uses
  the load multiple word instructions and the store multiple word
  instructions. Those instructions do not work when the processor is in
  little-endian mode (except PPC740/PPC750), so, we only enable it
  for big endian powerpc.

- for big endian ppc64, as the help message from arch/powerpc/Kconfig
  shows, the V2 ABI is standard for 64-bit little-endian, but for
  big-endian it is less well tested by kernel and toolchain, so, use
  elfv1 as-is, no need to explicitly ask toolchain to use elfv2 here.

[1]: https://qemu.readthedocs.io/en/latest/system/ppc/powernv.html
[2]: linuxppc/issues#402
[3]: https://lore.kernel.org/qemu-devel/20220504065536.3534488-1-aik@ozlabs.ru/

Suggested-by: Willy Tarreau <w@1wt.eu>
Link: https://lore.kernel.org/lkml/20230722121019.GD17311@1wt.eu/
Link: https://lore.kernel.org/lkml/20230719043353.GC5331@1wt.eu/
Reviewed-by: Thomas Weißschuh <linux@weissschuh.net>
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
Signed-off-by: Willy Tarreau <w@1wt.eu>
akiyks pushed a commit to akiyks/linux that referenced this issue Aug 24, 2023
Kernel uses ARCH=powerpc for both 32-bit and 64-bit PowerPC, here adds a
ppc64 variant for big endian 64-bit PowerPC, users can pass XARCH=ppc64
to test it.

The powernv machine of qemu-system-ppc64 is used with
powernv_be_defconfig.

As the document [1] shows:

  PowerNV (as Non-Virtualized) is the “bare metal” platform using the
  OPAL firmware. It runs Linux on IBM and OpenPOWER systems and it can be
  used as an hypervisor OS, running KVM guests, or simply as a host OS.

Notes,

- differs from little endian 64-bit PowerPC, vmlinux is used instead of
  zImage, because big endian zImage [2] only boot on qemu with x-vof=on
  (added from qemu v7.0) and a fixup patch [3] for qemu v7.0.51:

- since the VSX support may be disabled in kernel side, to avoid
  "illegal instruction" errors due to missing VSX kernel support, let's
  simply let compiler not generate vector/scalar (VSX) instructions via
  the '-mno-vsx' option.

- as 'man gcc' shows, '-mmultiple' is used to generate code that uses
  the load multiple word instructions and the store multiple word
  instructions. Those instructions do not work when the processor is in
  little-endian mode (except PPC740/PPC750), so, we only enable it
  for big endian powerpc.

- for big endian ppc64, as the help message from arch/powerpc/Kconfig
  shows, the V2 ABI is standard for 64-bit little-endian, but for
  big-endian it is less well tested by kernel and toolchain, so, use
  elfv1 as-is, no need to explicitly ask toolchain to use elfv2 here.

[1]: https://qemu.readthedocs.io/en/latest/system/ppc/powernv.html
[2]: linuxppc/issues#402
[3]: https://lore.kernel.org/qemu-devel/20220504065536.3534488-1-aik@ozlabs.ru/

Suggested-by: Willy Tarreau <w@1wt.eu>
Link: https://lore.kernel.org/lkml/20230722121019.GD17311@1wt.eu/
Link: https://lore.kernel.org/lkml/20230719043353.GC5331@1wt.eu/
Reviewed-by: Thomas Weißschuh <linux@weissschuh.net>
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
Signed-off-by: Willy Tarreau <w@1wt.eu>
Reset816 pushed a commit to Reset816/linux that referenced this issue Aug 31, 2023
Kernel uses ARCH=powerpc for both 32-bit and 64-bit PowerPC, here adds a
ppc64 variant for big endian 64-bit PowerPC, users can pass ARCH=ppc64
to test it.

The powernv machine of qemu-system-ppc64 is used with
powernv_be_defconfig.

As the document [1] shows:

  PowerNV (as Non-Virtualized) is the “bare metal” platform using the
  OPAL firmware. It runs Linux on IBM and OpenPOWER systems and it can be
  used as an hypervisor OS, running KVM guests, or simply as a host OS.

Notes,

- differs from little endian 64-bit PowerPC, vmlinux is used instead of
  zImage, because big endian zImage [2] only boot on qemu with x-vof=on
  (added from qemu v7.0) and a fixup patch [3] for qemu v7.0.51:

- since the VSX support may be disabled in kernel side, to avoid
  "illegal instruction" errors due to missing VSX kernel support, let's
  simply let compiler not generate vector/scalar (VSX) instructions via
  the '-mno-vsx' option.

- as 'man gcc' shows, '-mmultiple' is used to generate code that uses
  the load multiple word instructions and the store multiple word
  instructions. Those instructions do not work when the processor is in
  little-endian mode (except PPC740/PPC750), so, we only enable it
  for big endian powerpc.

[1]: https://qemu.readthedocs.io/en/latest/system/ppc/powernv.html
[2]: linuxppc/issues#402
[3]: https://lore.kernel.org/qemu-devel/20220504065536.3534488-1-aik@ozlabs.ru/

Suggested-by: Willy Tarreau <w@1wt.eu>
Link: https://lore.kernel.org/lkml/20230722121019.GD17311@1wt.eu/
Link: https://lore.kernel.org/lkml/20230719043353.GC5331@1wt.eu/
Reviewed-by: Thomas Weißschuh <linux@weissschuh.net>
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
charliemirabile pushed a commit to underground-software/nolibc that referenced this issue Jan 28, 2024
Kernel uses ARCH=powerpc for both 32-bit and 64-bit PowerPC, here adds a
ppc64 variant for big endian 64-bit PowerPC, users can pass XARCH=ppc64
to test it.

The powernv machine of qemu-system-ppc64 is used with
powernv_be_defconfig.

As the document [1] shows:

  PowerNV (as Non-Virtualized) is the “bare metal” platform using the
  OPAL firmware. It runs Linux on IBM and OpenPOWER systems and it can be
  used as an hypervisor OS, running KVM guests, or simply as a host OS.

Notes,

- differs from little endian 64-bit PowerPC, vmlinux is used instead of
  zImage, because big endian zImage [2] only boot on qemu with x-vof=on
  (added from qemu v7.0) and a fixup patch [3] for qemu v7.0.51:

- since the VSX support may be disabled in kernel side, to avoid
  "illegal instruction" errors due to missing VSX kernel support, let's
  simply let compiler not generate vector/scalar (VSX) instructions via
  the '-mno-vsx' option.

- as 'man gcc' shows, '-mmultiple' is used to generate code that uses
  the load multiple word instructions and the store multiple word
  instructions. Those instructions do not work when the processor is in
  little-endian mode (except PPC740/PPC750), so, we only enable it
  for big endian powerpc.

- for big endian ppc64, as the help message from arch/powerpc/Kconfig
  shows, the V2 ABI is standard for 64-bit little-endian, but for
  big-endian it is less well tested by kernel and toolchain, so, use
  elfv1 as-is, no need to explicitly ask toolchain to use elfv2 here.

[1]: https://qemu.readthedocs.io/en/latest/system/ppc/powernv.html
[2]: linuxppc/issues#402
[3]: https://lore.kernel.org/qemu-devel/20220504065536.3534488-1-aik@ozlabs.ru/

Suggested-by: Willy Tarreau <w@1wt.eu>
Link: https://lore.kernel.org/lkml/20230722121019.GD17311@1wt.eu/
Link: https://lore.kernel.org/lkml/20230719043353.GC5331@1wt.eu/
Reviewed-by: Thomas Weißschuh <linux@weissschuh.net>
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
Signed-off-by: Willy Tarreau <w@1wt.eu>
Upstream-commit: faeb4e09fe77262f9a6b2f9f874eec0b6850721e
charliemirabile pushed a commit to underground-software/nolibc that referenced this issue Jan 28, 2024
Kernel uses ARCH=powerpc for both 32-bit and 64-bit PowerPC, here adds a
ppc64 variant for big endian 64-bit PowerPC, users can pass XARCH=ppc64
to test it.

The powernv machine of qemu-system-ppc64 is used with
powernv_be_defconfig.

As the document [1] shows:

  PowerNV (as Non-Virtualized) is the “bare metal” platform using the
  OPAL firmware. It runs Linux on IBM and OpenPOWER systems and it can be
  used as an hypervisor OS, running KVM guests, or simply as a host OS.

Notes,

- differs from little endian 64-bit PowerPC, vmlinux is used instead of
  zImage, because big endian zImage [2] only boot on qemu with x-vof=on
  (added from qemu v7.0) and a fixup patch [3] for qemu v7.0.51:

- since the VSX support may be disabled in kernel side, to avoid
  "illegal instruction" errors due to missing VSX kernel support, let's
  simply let compiler not generate vector/scalar (VSX) instructions via
  the '-mno-vsx' option.

- as 'man gcc' shows, '-mmultiple' is used to generate code that uses
  the load multiple word instructions and the store multiple word
  instructions. Those instructions do not work when the processor is in
  little-endian mode (except PPC740/PPC750), so, we only enable it
  for big endian powerpc.

- for big endian ppc64, as the help message from arch/powerpc/Kconfig
  shows, the V2 ABI is standard for 64-bit little-endian, but for
  big-endian it is less well tested by kernel and toolchain, so, use
  elfv1 as-is, no need to explicitly ask toolchain to use elfv2 here.

[1]: https://qemu.readthedocs.io/en/latest/system/ppc/powernv.html
[2]: linuxppc/issues#402
[3]: https://lore.kernel.org/qemu-devel/20220504065536.3534488-1-aik@ozlabs.ru/

Suggested-by: Willy Tarreau <w@1wt.eu>
Link: https://lore.kernel.org/lkml/20230722121019.GD17311@1wt.eu/
Link: https://lore.kernel.org/lkml/20230719043353.GC5331@1wt.eu/
Reviewed-by: Thomas Weißschuh <linux@weissschuh.net>
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
Signed-off-by: Willy Tarreau <w@1wt.eu>
Upstream-commit: faeb4e09fe77262f9a6b2f9f874eec0b6850721e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug It's a bug enhancement Addition/modification of a feature, not a bug per se in-progress Some work has been done on this issue
Projects
Status: Stale
Development

No branches or pull requests

4 participants