Features

[bennuttall]

A feature-complete piwheels should:

• be able to build ARM platform wheels which work on Raspberry Pi
• be able to host wheels on a web server which are installable with pip
• be able to log the status and output of each build
• maintain an updated complete list of packages on PyPI, along with version numbers
• be able to rebuild packages when new versions are released
• be able to build ARMv7 wheels
• be able to provide ARMv6 wheels
• be able to build wheels for multiple Python versions
• be able to publicly provide output from failed build attempts
• be autonomous
• be resilient to failure

[lurch]

Are the build-logs publicly viewable? If so, a link to them from http://www.piwheels.hostedpi.com/ would be nice ;-)

[waveform80]

Resilience to failure we're already pretty good at. While the monitor crashed a few times in the latest run (though I think I've fixed that in fc2e95f), and the odd slave crashed when a huge C++ build wound up crashing the whole machine, the master itself didn't crash once in spite of the above failures.

@lurch build logs aren't currently publicly viewable but they have all been captured in the database. I'll see if I can include them in the JSON output envisaged for #16. If you want a copy in the meantime I can send you a dump of the postgres database (it's not terribly big - about 80Mb or so).

[waveform80]

Oh, sorry - make that 120Mb or so (hadn't checked since the latest run finished).

[lurch]

Ahhh, http://www.piwheels.hostedpi.com/ has changed since I wrote that comment ;-) IIRC at the time it was just a big directory-index of all the packages it had built (or perhaps I'm mis-remembering).

I was just thinking that for those packages that didn't get built successfully, it might be useful if the build-logs were publicly viewable (I was thinking in terms of a web-page rather than an API) so that package-maintainers could see what needed fixing, in order to get their package building properly on piwheels.

[waveform80]

Yeah, that huge list (required for pip) is now at http://www.piwheels.hostedpi.com/simple (WARNING: takes a long time to load in a browser) and you can see the indexes for individual packages beneath that, e.g. http://www.piwheels.hostedpi.com/simple/RPi.GPIO (worth noting if you check the source for the latter link that the links all include the SHA256 hash for download verification).

[lurch]

Hmmmm, https://www.piwheels.hostedpi.com/simple/RPi.GPIO/RPi.GPIO-0.6.3-cp34-cp34m-linux_armv7l.whl works, but https://www.piwheels.hostedpi.com/simple/RPi.GPIO/RPi.GPIO-0.6.3-cp34-cp34m-linux_armv6l.whl gives me a "Forbidden" error 😕

[bennuttall]

That's odd. ARMv6 wheels are symlinks to the ARMv7 files and they're owned by the same user.

[lurch]

Maybe there's a setting in Apache that prevents it serving symlinks?

[lurch]

P.S. Given that the ARMv6 wheels are a symlink to the wheels that have been built for ARMv7, it seems a bit disingenuous for https://www.piwheels.hostedpi.com/ to claim "At present, piwheels only builds wheels for Python 3.4 on the armv6l architecture .... although these wheels are duplicated for the armv7l architecture" ?
(There's still part of me that wonders if one of the ARMv7 wheels might contain ARMv7-only instructions that wouldn't be able to run on an ARMv6 Raspberry Pi?)

[waveform80]

P.S. Given that the ARMv6 wheels are a symlink to the wheels that have been built for ARMv7, it seems a bit disingenuous for https://www.piwheels.hostedpi.com/ to claim "At present, piwheels only builds wheels for Python 3.4 on the armv6l architecture .... although these wheels are duplicated for the armv7l architecture" ?
(There's still part of me that wonders if one of the ARMv7 wheels might contain ARMv7-only instructions that wouldn't be able to run on an ARMv6 Raspberry Pi?)

Ahhh, it's sort of disingenuous but in the opposite way to what you're thinking :) All the build slaves are Pi3's and therefore they think they're building ARMv7 wheels. However, my understanding is that currently everything's really armv6 and just appears as armv7 because the Raspbian kernel lies about its architecture. Hence the statement that we're building armv6 wheels and duplicating for armv7 (in as much as we're building armv6 wheels that are labelled armv7 and then using symlinks to relabel them as armv6 ... but I didn't feel like explaining that in the webpage! :).

[lurch]

However, my understanding is that currently everything's really armv6 and just appears as armv7 because the Raspbian kernel lies about its architecture.

Huh? I thought that's why there were two kernel images - kernel.img for ARMv6 and kernel7.img for ARMv7 ? *confused*

[bennuttall]

Huh? I thought that's why there were two kernel images - kernel.img for ARMv6 and kernel7.img for ARMv7 ? confused

Correct. I think that's just for booting, but the OS runs in ARMv6 mode.

FWIW I think the wording is misleading. It's a lie mitigating a different lie. But I think it could be better phrased. I have a redraft I'll push later.

[lurch]

Correct. I think that's just for booting, but the OS runs in ARMv6 mode.

Perhaps you could ask some of the RPF engineers for clarification? ;-)

[waveform80]

Correct. I think that's just for booting, but the OS runs in ARMv6 mode.

Sorry, that's what I meant - that the userland is still ARMv6 and everything built in the userland is still ARMv6. But pip (or certain bits of pip, that ultimately lead back to Python's platform module) query uname to determine what architecture they're on; that reports armv7l ... which might be true for the kernel but is wrong for everything else, and lo ... my explanations descend into chaos! Anyway, the wording on the homepage is indeed mis-leading (deliberately to some degree) and whatever clarity anybody can offer is most welcome indeed :)

[lurch]

Aha, that's what I understood - ARMv7 kernel with ARMv6 userland. But if pip is using uname to query the arch to compile for, could it still end up with ARMv7-specific code inside the wheels? Or is the ARMv6 version of gcc unable to generate ARMv7-specific object code? *shrug*

[waveform80]

My understanding, limited as it is, is that the userland is still strictly ARMv6 and thus gcc will still generate ARMv6-compatible code - hence why we're happy to symlink wheels that are ostensibly ARMv7 to their equivalent ARMv6 name and serve them up as such. This is admittedly a lie and not good for any platform which is strictly ARMv7 only (i.e. any ARMv7 non-pi platform), but I'm hopeful the name "piwheels" makes it pretty clear this is a Pi-specific service (of course, it doesn't help anyone dealing with "armv7" wheels built by Pi users and registered on PyPI :)

[lurch]

AIUI a strictly ARMv6-only piwheel would work fine on a strictly ARMv7-only platform (ARMv7 is backwards-compatible with ARMv6), but the inverse may not be true. I.e. a ARMv7-only piwheel might not run on a ARMv6-only platform (like Raspbian running on a Pi1) if it contains any ARMv7-specific instructions. But I believe (which might be an incorrect assumption!) in the majority of cases, most ARMv7-code probably doesn't contain any ARMv7-specific instructions, and so will still run fine on an ARMv6 CPU?

[waveform80]

That's also my understanding, architecturally for the Pi (though I'm also given to understand that non-Pi ARMv7 platforms aren't necessarily ARMv6 compatible). I'm also under the impression, perhaps mistakenly, that currently anything compiled on Raspbian on a v7 platform (say, a Pi 2) is v6 compatible as the whole userland (including compiler and libs) is v6. Happy to be corrected about any of the above!

And if anyone wants a crack at refining the notes on the homepage, the relevant file is piwheels/master/index.template.html. If you're wondering about the {{ and }} it's because the whole thing gets chucked through Python's str.format call to fill in the stats, so bear that in mind for any patches.

[lurch]

I'm also under the impression, perhaps mistakenly, that currently anything compiled on Raspbian on a v7 platform (say, a Pi 2) is v6 compatible as the whole userland (including compiler and libs) is v6.

I don't have time to test / investigate that myself, but again perhaps it'd be best for @bennuttall to ask some of the RPF engineers about that, as they're obviously the ones most likely to know the correct answers! ;-)

[bennuttall]

Almost all the items are complete now. @waveform80 do you think it's possible to share build output, perhaps via the prposed API rather than via web pages as originally suggested?

[waveform80]

Sure - I can add that pretty quickly as an extension to the index_scribe task (whenever it's prodded to rebuild the index for a package it queries everything it needs from the database anyway so that's an ideal time to grab other stuff like build output and dump it all in there).

Just one thing to bear in mind: the build output is large (it's the bulk of the database, and that's with pg compressing it all via TOAST) and JSON is not exactly an efficient format. I wouldn't be surprised if adding this chews up about 10Gb of drive space which is about all we've got left at the moment. Might be worth expanding the master's storage before we do this.

[bennuttall]

Closed by #298 🥳