nxdk has no proper libc

[JayFoxRox]

• printf is absolutely horrible and doesn't respect most format specifiers (not Xbox specific).
  A better minimalistic alternative is even present in cromwell
• fopen and friends are missing (Xbox specific).
• I'd appreciate C11 thread compatibility too (Xbox specific).

I'd like to hear about peoples opinion which libc we should use or if we should implement our own.
I have never ported a libc and I'm not sure how C++ compatibility would play into this in the future.

I'm also not sure how a libc would handle the platform specific parts in contrast to the non-platform specific parts (such as the printf stuff)

However, this is an important issue which causes a lot of trouble in any Xbox related homebrew I have.

[thrimbor]

Maybe this is the right time to talk about what I've been doing.
Around the same time as nxdk was started, I started doing my own XDK, but instead of basing it on OpenXDK, I started from scratch. It's far from being complete, but it has the following features:

1. A 100% complete xboxkrnl.h (related to Update kernel imports #14). It's monolithic (but could be split up), not fully documented (large parts of it are documented using Doxygen) and may have a constant or two missing, but when it comes to the function declarations, I'm pretty confident it's complete and correct.
2. An xboxkrnl.lib built from "source" (i.e. a definition-file)
3. A libc. It's far from being complete (fopen isn't working yet, printf isn't implemented) and may not be bug-free, but it also has some improvements over what nxdk is currently using:
   1. Correct string and memory comparison. The code you took from lib43 may be working for you, but it's not fulfilling the requirements the specification states.
   2. Far more math.h functions.
   3. More efficient memory allocation (nxdk's malloc always reserves complete 4KiB pages which I consider wasteful considering memory is pretty tight on the XBox)
   4. The ability to use a standards-compliant main() function
   5. C11 threads, mutexes and call_once.

Missing parts like printf could get picked from the CC0-licensed "pdclib", which some of my code is already based on.

There are several more things I worked on or experimented with like an OpenAL-implementation with some tricks to make it run faster than existing ones, C++ support and support for thread-specific storage (which is especially tricky because cxbe simply ignores TSS).

However, there are reasons why I didn't already contribute to nxdk or base my XDK on OpenXDK code.

1. License problems: OpenXDK is GPLv2 licensed, pbkit is AFL. Those two are considered incompatible by the FSF, which might make distribution of binaries produced with nxdk illegal! I prefer the GPL for my own personal projects and certainly think that all tools involved in building an XBox-binary should be GPL-licensed, but I don't think it's the right choice for an XDK. I have not chosen a license for my own project yet, but I think GPLv3+LE (which permits creating proprietary binaries but still puts the XDK under GPLv3) could be a good choice. However, it would still be incompatible with pbkit, unless openxdkman agrees to re-license it.
2. The buildsystem of nxdk. Having to pull in nxdk source-dependencies into the Makefile is unusual and seems pretty messy to me. My own XDK is built separately and merges all libraries into one big library file which can then be linked against. That also makes it possible to have easier-to-use binary releases.

Since there is little hope that I can find enough time to make my own project as capable as nxdk currently is. I am interested in joining forces. There my be need for a discussion if and how this can be done, however.

[JayFoxRox]

Sounds good. I hope you put your code online soon.

---

An xboxkrnl.lib built from "source" (i.e. a definition-file)

clarification: so is ours, but the one created by mingw does not work, I'm not sure how to do it with clang and I don't want to run Windows just to generate a new lib.

fopen isn't working yet

This one bothers me in nxdk and I'll probably tackle it soon out of frustration. OpenXDK has a working implementation which I'd steal temporarily but with some bugfixes.
We can later iterate to create our own / better code. I've just looked at pdclib and they have a win32 port which should pretty much work out of the box

More efficient memory allocation (nxdk's malloc always reserves complete 4KiB pages which I consider wasteful considering memory is pretty tight on the XBox)

Doesn't NtVirtualAlloc already take care of that? Maybe I'm mistaken though. I never really looked into the MS memory allocator
Consider creating a seperate issue about this

experimented with like an OpenAL-implementation

(My response has moved to #20 )

C++ support

Very interesting if you can get that to work somehow

support for thread-specific storage (which is especially tricky because cxbe simply ignores TSS)

This should be easily fixable (I think).
But again: probably not necessary for my use cases as my threading currently happens in nxdk-rdt or python scripts where TLS is done manually.

License problems: OpenXDK is GPLv2 licensed, pbkit is AFL.

(My response has moved to #21 )

Having to pull in nxdk source-dependencies into the Makefile is unusual and seems pretty messy to me. My own XDK is built separately and merges all libraries into one big library file which can then be linked against.

Agreed. I'm super annoyed by the Makefiles anyway and I'd have preferred CMake, let alone the cluttered *.c and *.c.d files.
I also prefer your / OpenXDK approach of having a handful of libs to link against.
(but temporarily, the current approach is fine with me - fixing this is low-priority to me)

---

My IRC bouncer is currently down, but it would be nice if you can join #xqemu on freenode.
I'm also available on the cxbx-reloaded gitter.im and the Xbox emulation discord (temporarily, I intend to leave discord soon)
Contributions to http://xboxdevwiki.net would also be welcome! You'll find / get the anti-spam registration password in any of the previously mentioned channels

[mborgerson]

Doesn't NtVirtualAlloc already take care of that? Maybe I'm mistaken though. I never really looked into the MS memory allocator

I was under the impression that NtVirtualAlloc was a general purpose allocate (sub-page level) too. Could be wrong. VirtualAlloc documentation mentions allocation size is rounded-up to page boundary. Would need to confirm on Xbox side if behavior is the same.

(regarding audio, post is now in #20 )
To me, writing these kind of libs is not a priority to be honest, but be my guest. The truth is that most users of nxdk don't intend to write / port much homebrew.

Agree. @thrimbor, if you want to do the work to make it better, by all means. By the way, IIRC OpenXDK got sound working with SDL. Might be a good place to start, but YMMV.

C++ support
Very interesting if you can get that to work somehow

Simple C++ support is not hard to add. You just need to handle some of the C++ ness (make sure you call the static (de-)constructors appropriately). I think the STL stuff will be handled by the compiler and doesn't need any special support. Object instantiation (via new) probably needs to be hooked up to malloc/free.

Agreed. I'm super annoyed by the Makefiles anyway and I'd have preferred CMake, let alone the cluttered *.c and *.c.d files. I also prefer your / OpenXDK approach of having a handful of libs to link against. (but temporarily, the current approach is fine with me - fixing this is low-priority to me)

What's with all the Makefile hate? 😄 As for the clutter, we can move the build artifacts (*.d, *.o, and friends) into a "build" directory instead of having them sit next to their source counterparts. -- Now that NXDK has grown to include a lot more functionality, I think the libraryization of things makes sense, and would be cleaner...if someone wants to do the work for any of this (I'm not bothered enough by it at the moment to invest the time).

My IRC bouncer is currently down

Coincidentally, my IRC bouncer is also down.. for the last month or so. I get the GitHub e-mails about comments, so I can usually respond to those within a couple days, depending on content and if I'm not too busy. @JayFoxRox I like IRC, but maybe setting up a gitter.im for XQEMU would lower the barrier of entry for new developers... thoughts?

[thrimbor]

I was under the impression that NtVirtualAlloc was a general purpose allocate (sub-page level) too. Could be wrong.

Unfortunately not, it doesn't provide a heap, it's really just allocation of pages for the virtual address space. That's also why it allows you to specify pagetable flags ;)

Quick & dirty test:

for (int r=0; r<20; r++)
        DbgPrint("0x%08x\n", malloc(1));

Xbox image loaded: \Device\Harddisk0\Partition1\devkit\malte\MALTE.XBE
0x00040000
0x00050000
0x00060000
0x00070000
0x00080000
0x00090000
0x000a0000
0x000b0000
0x000c0000
0x000d0000
0x000e0000
0x000f0000
0x00100000
0x00110000
0x00120000
0x00130000
0x00140000
0x00150000
0x00160000
0x00170000

A full page for every one-byte allocation.

Simple C++ support is not hard to add.

I was working on porting libc++, which has quite a few requirements to work correctly. It has been a while since I worked on this, and I put it aside when I couldn't find the time to investigate name-mangling issues.

I might work on bringing over my kernel headers and libc to NXDK in the next few days, but that depends on whether I can find the time.

[ghost]

I think my code with xexec leaves potential for a cross-between POSIX & WinNT API translation library, which would make nxdk work better.

All XBEs I've come across have their own libc routines embedded. They are really static built executables. The libc library is included in each binary build.

For thread compatibility, definitely pthread.

This is a good idea.

[mborgerson]

Can you elaborate on how api translation would make nxdk work better? xexec does kernel API to POSIX, but nxdk builds executables targeting Xbox and kernel API so you would want to go the other direction.

That's right, with exception to xbe kernel imports and any custom dynamic linking stuff (xbmc does this for instance), all libs are statically linked.

My preference for nxdk is to keep things as simple as possible and to not introduce any hard dependencies. That said, I'm open to considering a better libc library, especially a better printf (which there are many even public domain implementations). I plan on adding one eventually, but would happily merge a PR if someone wants to do it.

If someone wants to add some library for POSIX support (eg pthreads), that would be great too. PRs welcome.

[Grovespaz]

I want to take a crack at porting newlib. Before I dive too deep into this, does anyone know of any reason why our libc shouldn't be newlib for the foreseeable future?