devkitarm-r52 WIP. by orbea · Pull Request #8772 · libretro/RetroArch

orbea · 2019-05-12T19:39:04Z

Description

I am making this as PR because it is easier to show what has been done with diffs. This should not be merged as is and needs further discussion first.

After a lot of effort I managed to build and install devkitarm-r52 using their buildscripts repo. It is complete with the toolchain, SDKs and tools. It seems that RetroArch has not been fully updated for the new release and upstream suggests only using new releases.

For reference here are the list of files in my package.

devkitarm.txt

I am sourcing this script before building anything.

# set needed variables
export DEVKITPRO=/opt/devkitpro
export DEVKITARM=${DEVKITPRO}/devkitARM
export PATH="${DEVKITPRO}/tools/bin:$PATH"

. /etc/profile.d/devkitarm.sh

And before I built RetroArch I built without issue gambatte and copied gambatte_libretro_ctr.a to the RetroArch source directory as libretro_ctr.a.

make -f Makefile.libretro platform=ctr

Related Issues

When building RetroArch I encountered a few issues.

When compiling Makefile.ctr.salamander I found that mcuHwcGetBatteryLevel is now MCUHWC_GetBatteryLevel.
It appears several tools moved from $(DEVKITARM)/bin to ${DEVKITPRO}/tools/bin. The upstream buildscripts suggests adding this directory to the $PATH. I think these probably should be not be absolute paths.
As seen in the following commit gfxSetFramebufferInfo is now static and I do not think we can use it. I just got undefined references during linking.

devkitPro/libctru@4ceebb8

The minimal changes I made in the example commit allow the build to succeed, but I do not have an environment to test the builds and I do not expect them to be correct.

Reviewers

@jdgleaver Maybe you can help me understand what is going on here? Also can you explain the process for building multiple cores? The RetroArch docs do not explain that other than suggesting to use libretro-super scripts.

jdgleaver · 2019-05-13T09:28:59Z

@orbea Are you volunteering to take sole responsibility for updating, testing and maintaining the 3DS port? Because that's what this PR entails. Not trying to stop you or anything - just want to make sure you know what you're getting into!

This represents a colossal leap in the version of ctrulib. You'll have to go through all the CTR gfx and platform code with a fine tooth comb (and this is not just a matter of getting it to compile - you'll have to go through our custom stuff line-by-line, and check that it tallies with the new libraries).

I have also seen reports that newer versions of ctrulib have issues using the enhanced features of n3DS, which would be a deal breaker. You'll have to verify that this has been fixed, or provide a workaround.

Once everything is building correctly. you'll need to test every core on both a (real) o3DS and n3DS, performing a careful performance/stability analysis to ensure there are no regressions. With the recent 1.7.7 release this is incredibly important, since we finally (after several years) have a polished/performant 3DS port that we can all be proud of. Any back-steps would suck.

I should also mention that this version of devkitarm is not compatible with the OS/toolchains I use, so I won't be able to help with any of this, nor will I be able to work on 3DS development in the future. Again, I'm not trying to stop you here, but it's maybe something you should be aware of.

Anyway, you asked how to build cores - it's very easy:

> cd /path/to/libretro-super
> ./libretro-fetch.sh <core_name_1> <core_name_2> <core_name_3> <etc>
> ./libretro-build-ctr.sh <core_name_1> <core_name_2> <core_name_3> <etc>
> cp -v dist/ctr/* /path/to/RetroArch/dist-scripts/
> cp /path/to/RetroArch/dist-scripts/
> ./dist-cores.sh ctr

Everything will end up in RetroArch/pkg/ctr/build/

orbea · 2019-05-13T13:04:14Z

@jdgleaver My goal is to make these console builds more reproducible and eventually try adding them to travis so that we can prevent build failures before they happen. The existing libretro documentation is lacking and doesn't even explain what toolchain version to use. Frankly there are too many issues in constantly backtracking to find regressions when there too many broken commits to bisect (I just happened to start with devkitarm and 3ds).

Additionally it makes it hard fixing something things in the build process when I can't test these console builds.

However it seems clear that fixing this will be a huge job and my motivation is entirely gone when having to deal with hacked up and outdated toolchains which will become only less practical in the future. And as I lack hardware to fully test this that is another barrier...

I should also mention that this version of devkitarm is not compatible with the OS/toolchains I use,

Can you elaborate? Making it hard for you to help is certainly not my goal. :)

Anyway, you asked how to build cores - it's very easy:

I was asking how to do it manually without libretro-super, I suppose I may have to read the scripts to understand what they are doing.

jdgleaver · 2019-05-13T13:50:04Z

You may have gotten the wrong impression from my comment. I was in no way trying to discourage you - just pointing out the difficulties :)

My goal is to make these console builds more reproducible and eventually try adding them to travis so that we can prevent build failures before they happen. The existing libretro documentation is lacking and doesn't even explain what toolchain version to use. Frankly there are too many issues in constantly backtracking to find regressions when there too many broken commits to bisect (I just happened to start with devkitarm and 3ds).

This is a noble goal. I understand completely what you're saying.

Imagine how hard is was for me - a complete outsider at the time - to find the correct version of the 3DS toolchain hidden away on the buildbot. Complete detective work, with no help whatsoever. This is not good for the project...

However: Some 3DS builds are already on travis, and they work fine - maybe that part of the job just involves making sure that all the supported 3DS cores are on travis...?

Additionally it makes it hard fixing something things in the build process when I can't test these console builds.

That's something that can only be fixed by owning hardware... :(

Could you maybe ask for some Patreon funds for this sort of thing? I mean, you're kinda important, and if you can't do your job then that's a problem for everyone...

However it seems clear that fixing this will be a huge job and my motivation is entirely gone when having to deal with hacked up and outdated toolchains which will become only less practical in the future.

Well... I don't know if it's any help, but AFAIK devkitPro has been discontinued (so there will be no more updates). It is therefore not unreasonable for us to choose/fork a version for our own use...

Can you elaborate? Making it hard for you to help is certainly not my goal. :)

The 'official' devkitPro installation is pure arse.

I remember I spent a whole week trying to get the build to work, but ultimately it's not compatible with my version of gcc. I use my computers for other (more important) things that have specific requirements, so updating gcc is not something I'm prepared to do.

I am very good at hacking things and implementing workarounds, but the latest devkitPro was just too much crap to deal with...

I was asking how to do it manually without libretro-super, I suppose I may have to read the scripts to understand what they are doing.

Oh right - sorry, didn't realise that.

In general, you can just use the environment setup you've already got, plus an extra CTRULIB variable. So that's:

# set needed variables
export DEVKITPRO=/opt/devkitpro
export DEVKITARM=${DEVKITPRO}/devkitARM
export PATH="${DEVKITPRO}/tools/bin:$PATH"
export CTRULIB="${DEVKITPRO}/libctru"

Then just go to your repo directory and do:

> export platform=ctr
> make

You may run into some oddities along the way, but that's mostly all there is to it.

orbea · 2019-05-13T14:15:56Z

You may have gotten the wrong impression from my comment. I was in no way trying to discourage you - just pointing out the difficulties :)

I understand and entirely appreciate you being frank with me. :)

Imagine how hard is was for me - a complete outsider at the time - to find the correct version of the 3DS toolchain hidden away on the buildbot. Complete detective work, with no help whatsoever. This is not good for the project...

However: Some 3DS builds are already on travis, and they work fine - maybe that part of the job just involves making sure that all the supported 3DS cores are on travis...?

Yes, I can imagine! I've procrastinated on starting this so long for a reason...

I was thinking RetroArch in particular, the cores could probably come second and there are problems with using libretro-super with travis that are very tedious to debug and test. I have had issues with both the error codes (travis would pass with failed builds) and getting useful logs in the past.

Well... I don't know if it's any help, but AFAIK devkitPro has been discontinued (so there will be no more updates). It is therefore not unreasonable for us to choose/fork a version for our own use...

With all due honesty their documentation is lacking to the point of being non-existent, but this is the first I have heard about this. Do you have a source? Fwiw when building this I saw no indication of this in their git repos. Here are the sources I used, most of these tarballs are rather recent.

3dslink-0.5.2.tar.bz2           gp32-tools-1.0.3.tar.bz2
3dstools-1.1.4.tar.bz2          grit-0.8.15.tar.bz2
binutils-2.32.tar.xz            libctru-src-1.5.1.tar.bz2
buildscripts-20190329.tar.bz2   libfat-src-1.1.3.tar.bz2
citro2d-src-1.1.0.tar.bz2       libfilesystem-src-0.9.14.tar.bz2
citro3d-src-1.5.0.tar.bz2       libgba-src-0.5.2.tar.bz2
default-arm7-src-0.7.4.tar.bz2  libmirko-src-0.9.8.tar.bz2
devkitarm-crtls-1.0.0.tar.xz    libnds-src-1.7.2.tar.bz2
devkitarm-rules-1.0.0.tar.xz    maxmod-src-1.0.11.tar.bz2
dfu-util-0.9.1.tar.bz2          mmutil-1.8.7.tar.bz2
dstools-1.2.1.tar.bz2           ndstool-2.1.1.tar.bz2
dswifi-src-0.4.2.tar.bz2        newlib-3.1.0.tar.gz
gba-tools-1.1.0.tar.bz2         picasso-2.7.0.tar.bz2
gcc-8.3.0.tar.xz                stlink-1.2.3.tar.bz2
gdb-8.2.1.tar.xz                tex3ds-1.0.0.tar.bz2
general-tools-1.0.2.tar.bz2

DOWNLOAD="https://github.com/devkitPro/3dslink/releases/download/v0.5.2/3dslink-0.5.2.tar.bz2 \
          https://github.com/devkitPro/3dstools/releases/download/v1.1.4/3dstools-1.1.4.tar.bz2 \
          https://github.com/devkitPro/buildscripts/releases/download/sources/binutils-2.32.tar.xz \
          https://github.com/devkitPro/buildscripts/releases/download/v20190329/buildscripts-20190329.tar.bz2 \
          https://github.com/devkitPro/citro2d/releases/download/v1.1.0/citro2d-src-1.1.0.tar.bz2 \
          https://github.com/fincs/citro3d/releases/download/v1.5.0/citro3d-src-1.5.0.tar.bz2 \
          https://github.com/devkitPro/default-arm7/releases/download/v0.7.4/default-arm7-src-0.7.4.tar.bz2 \
          https://github.com/devkitPro/devkitarm-crtls/releases/download/v1.0.0/devkitarm-crtls-1.0.0.tar.xz \
          https://github.com/devkitPro/devkitarm-rules/releases/download/v1.0.0/devkitarm-rules-1.0.0.tar.xz \
          https://github.com/devkitPro/buildscripts/releases/download/sources/dfu-util-0.9.1.tar.bz2 \
          https://github.com/devkitPro/dstools/releases/download/v1.2.1/dstools-1.2.1.tar.bz2 \
          https://github.com/devkitPro/dswifi/releases/download/v0.4.2/dswifi-src-0.4.2.tar.bz2 \
          https://github.com/devkitPro/gba-tools/releases/download/v1.1.0/gba-tools-1.1.0.tar.bz2 \
          https://github.com/devkitPro/buildscripts/releases/download/sources/gcc-8.3.0.tar.xz \
          https://github.com/devkitPro/buildscripts/releases/download/sources/gdb-8.2.1.tar.xz \
          https://github.com/devkitPro/general-tools/releases/download/v1.0.2/general-tools-1.0.2.tar.bz2 \
          https://github.com/devkitPro/gp32-tools/releases/download/v1.0.3/gp32-tools-1.0.3.tar.bz2 \
          https://github.com/devkitPro/grit/releases/download/v0.8.15/grit-0.8.15.tar.bz2 \
          https://github.com/smealum/ctrulib/releases/download/v1.5.1/libctru-src-1.5.1.tar.bz2 \
          https://github.com/devkitPro/libfat/releases/download/v1.1.3/libfat-src-1.1.3.tar.bz2 \
          https://github.com/devkitPro/libfilesystem/releases/download/v0.9.14/libfilesystem-src-0.9.14.tar.bz2 \
          https://github.com/devkitPro/libgba/releases/download/v0.5.2/libgba-src-0.5.2.tar.bz2 \
          https://github.com/devkitPro/libmirko/releases/download/v0.9.8/libmirko-src-0.9.8.tar.bz2 \
          https://github.com/devkitPro/libnds/releases/download/v1.7.2/libnds-src-1.7.2.tar.bz2 \
          https://github.com/devkitPro/maxmod/releases/download/v1.0.11/maxmod-src-1.0.11.tar.bz2 \
          https://github.com/devkitPro/mmutil/releases/download/v1.8.7/mmutil-1.8.7.tar.bz2 \
          https://github.com/devkitPro/ndstool/releases/download/v2.1.1/ndstool-2.1.1.tar.bz2 \
          https://github.com/devkitPro/buildscripts/releases/download/sources/newlib-3.1.0.tar.gz \
          https://github.com/fincs/picasso/releases/download/v2.7.0/picasso-2.7.0.tar.bz2 \
          https://github.com/devkitPro/buildscripts/releases/download/sources/stlink-1.2.3.tar.bz2 \
          https://github.com/devkitPro/tex3ds/releases/download/v1.0.0/tex3ds-1.0.0.tar.bz2"

I worry that maintaining a hard-forked toolchain with our manpower and workload in is just not practical or maintainable in the long term.

The 'official' devkitPro installation is pure arse.

I remember I spent a whole week trying to get the build to work, but ultimately it's not compatible with my version of gcc. I use my computers for other (more important) things that have specific requirements, so updating gcc is not something I'm prepared to do.

I am very good at hacking things and implementing workarounds, but the latest devkitPro was just too much crap to deal with...

I know what you mean, it took me a few days to get my package right, but at this point it should not be hard for me to update my build process in the future. I do have gcc-9.1.0 though and am not too daunted by finding a new compiler broke things (Which it did...). What gcc version do you have? I was considering trying to support this with gcc-5.3.0 which is in Slackware stable. I could potentially help with any build issues you have.

I missed the CTRULIB environment variable, but I as able to figure out how to build cores. What I am not sure is how build RetroArch with more than one core? I can add a single core archve to RetoArch and build it, but how about more than one?

jdgleaver · 2019-05-13T15:01:45Z

With all due honesty their documentation is lacking to the point of being non-existent, but this is the first I have heard about this. Do you have a source? Fwiw when building this I saw no indication of this in their git repos. Here are the sources I used, most of these tarballs are rather recent.

Apparently there was a lot of nastiness that left the lead dev suicidal: https://mobile.twitter.com/davejmurphy/status/1114955918549102593

I just checked, and maybe he's getting help, but I'm not sure that future development can be counted on... :(

I would maybe advocate a more balanced approach: Certain newer devices (e.g, Switch) are under active development, and keeping an up-to-date toolchain is important. But other consoles are at end of life (or way past end of life), and nothing is ever going to change with them. So there is no maintenance load in these cases - it's not like a firmware update is going to break anything, and cores in general don't use anything device specific (i.e. I cannot think that any working core will ever receive upstream changes that require a newer version of e.g. the 3DS toolchain).

I know it's not quite so simple as that, but there is something to be said for choosing your battles.

Just airing some thoughts here - you're probably in a better place to know what's actually best, though.

What gcc version do you have? I was considering trying to support this with gcc-5.3.0 which is in Slackware stable. I could potentially help with any build issues you have.

I'm on gcc (SUSE Linux) 4.8.5.

You know, I guess I probably could get the damn thing to compile if it were life or death (heck, I had to reverse engineer decade-old proprietary kernel drivers and get them working on modern systems at my last job...) but it's a lot of work - and then there's that great wall of RetroArch CTR stuff to update/test at the end of it.

I've already promised to implement a new thumbnail downloader and content scanner, and there just doesn't seem to be any time left these days... :(

orbea · 2019-05-13T16:28:55Z

Apparently there was a lot of nastiness that left the lead dev suicidal: https://mobile.twitter.com/davejmurphy/status/1114955918549102593

I just checked, and maybe he's getting help, but I'm not sure that future development can be counted on... :(

That is disconcerting, I hope he's getting whatever help needed.

My main concern is that there may come a time when these when old toolchains no longer build or work on newer systems. I am not advocating we always use the latest, just that we use one compatible with upstream sources. Ideally it should have minimal or upstreamable changes which could be applied as a diff and then occasionally updated. It seems this might be a good time to investigate updating the toolchain, but of course if you can't reproduce it then this is a no go. Unfortunately gcc 4.8.5 could make this difficult...is it at all possible to install a newer gcc in /opt which would not interfere with other projects?

jdgleaver · 2019-05-13T16:47:43Z

My main concern is that there may come a time when these when old toolchains no longer build or work on newer systems. I am not advocating we always use the latest, just that we use one compatible with upstream sources. Ideally it should have minimal or upstreamable changes which could be applied as a diff and then occasionally updated.

Okay, this is a fair point.

We currently rely on a bunch of old pre-compiled binaries, and that is quite dodgy (who knows when a future OS won't be able to run them any more...)

It seems this might be a good time to investigate updating the toolchain, but of course if you can't reproduce it then this is a no go. Unfortunately gcc 4.8.5 could make this difficult...is it at all possible to install a newer gcc in /opt which would not interfere with other projects?

Well I tell you what - I'm going to have to look at some OS updates fairly soon anyway (super painful, but it's got to be done at some point). I guess I can bring this forward a bit, and then gcc won't be an issue (for RetroArch, at least...). Can you wait until, say, the end of next month-ish? It'll be a big job for me, but once my other things are sorted we can have a look at updating the 3DS toolchain. Does that sound okay...?

orbea · 2019-05-13T17:15:20Z

Well I tell you what - I'm going to have to look at some OS updates fairly soon anyway (super painful, but it's got to be done at some point). I guess I can bring this forward a bit, and then gcc won't be an issue (for RetroArch, at least...). Can you wait until, say, the end of next month-ish? It'll be a big job for me, but once my other things are sorted we can have a look at updating the 3DS toolchain. Does that sound okay...?

Of course. I am more than happy to work with whatever schedule you can manage, its not something that needs to be finished asap, just it would be nice to do sometime in the future. Please ping me with whatever I can help with. I made this PR mostly for discussion and not merging. :)

fpscan · 2019-05-13T23:15:57Z

Looking forward to this commit

orbea · 2019-05-14T06:12:54Z

Just don't expect anything too soon, this will be a big project which is only in the early planning and discussion phase. :)

jdgleaver · 2019-05-14T08:47:12Z

@orbea Okay, we'll go with that then. I'll try to get the whole thing working (and tested!) at my end, and ping you when it's done. Then you can look at updating the toolchain on the buildbot, and integrating travis etc. :)

devkitarm-r52 WIP.

61be513

orbea closed this May 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Comments

devkitarm-r52 WIP.#8772

devkitarm-r52 WIP.#8772
orbea wants to merge 1 commit intolibretro:masterfrom
orbea:ctr

orbea commented May 12, 2019 •

edited

Loading

Uh oh!

jdgleaver commented May 13, 2019

Uh oh!

orbea commented May 13, 2019

Uh oh!

jdgleaver commented May 13, 2019

Uh oh!

orbea commented May 13, 2019 •

edited

Loading

Uh oh!

jdgleaver commented May 13, 2019

Uh oh!

orbea commented May 13, 2019

Uh oh!

jdgleaver commented May 13, 2019

Uh oh!

orbea commented May 13, 2019

Uh oh!

fpscan commented May 13, 2019

Uh oh!

orbea commented May 14, 2019

Uh oh!

jdgleaver commented May 14, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Comments

Conversation

orbea commented May 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Reviewers

Uh oh!

jdgleaver commented May 13, 2019

Uh oh!

orbea commented May 13, 2019

Uh oh!

jdgleaver commented May 13, 2019

Uh oh!

orbea commented May 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jdgleaver commented May 13, 2019

Uh oh!

orbea commented May 13, 2019

Uh oh!

jdgleaver commented May 13, 2019

Uh oh!

orbea commented May 13, 2019

Uh oh!

fpscan commented May 13, 2019

Uh oh!

orbea commented May 14, 2019

Uh oh!

jdgleaver commented May 14, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

orbea commented May 12, 2019 •

edited

Loading

orbea commented May 13, 2019 •

edited

Loading