Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mips32r2 #203

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Mips32r2 #203

wants to merge 8 commits into from

Conversation

cm8
Copy link

@cm8 cm8 commented Jun 24, 2018

Produces usable results, but for some hardcoded values (e.g. root path of toolchain) it may be desirable to supply the value on the command line, i.e. python3 -m kiwixbuild --target-platform mips32r2_uclibc_glibcxx_dyn --toolchain-root /path/to/mips/toolchain/.

If other mips toolchains deviate a lot in their path and binary name layout, however, then it may make more sense to create yet more (sub)classes in mips32r2.py to support these. This will be a future choice to make for people trying builds with openwrt toolchains.

It should be safe to assume that people experienced enough to build their own mips toolchains, can find rootpath variable in mips32r2.py and re-set that accordingly. When building with apt-get-able (debian default) mips toolchains this should not be necessary.

cm8 added 6 commits June 19, 2018 17:56
The target mips32r2_dyn is tested to fully compile, but untested to
run.  Target mips32r2_static reaches kiwix-lib and fails linking
pthread.  For further details see the comments in #48.

Some known problems:

* hardcoded values of icudt58l in some dependent packages
  unnoticable during compile, but will affect runtimes

* the libdir name problem
  x86_64-linux-gnu <> mips-linux-gnu

* pthread linking problem when compiling kiwix-lib statically
  may be a meson issue, since it somewhat claims to properly
  handle pthread linking in cross-compile situations; there
  are some url links in #48 laying out a proper workaround.
ATM the uclibc toolchain buildable by freetz (see
https://github.com/freetz/freetz) is used.  When
configuring freetz make sure

FREETZ_LIB_libuClibc__WITH_WCHAR=y
FREETZ_BUILD_TOOLCHAIN=y

are set, so uClibc++ as part of the toolchain is built with wchar_t
support.  Eventually root_path definition in mips32r2.py (hardcoded
for now) needs to be adjusted.  Some (all?) prebuilt, downloadable
tcs of the freetz project do not have wchar support in uClibc++ (but
uclibc does).

KNOWN PROBLEMS:

xapian-core currently does not compile with uClibc++
added
  mips32r2_uclibc_gclibcxx_dyn,  (preferred atm, tested on prod hw)
  mips32r2_uclibc_gclibcxx_static
targets (shared) and renamed previous uclibc ones to
  mips32r2_uclibc_uclibcxx_dyn,
  mips32r2_uclibc_uclibcxx_static
for clarity (i.e. non-shared, pure ones)

reworked builder classes to use inheritance of properties and
methods (instead of copying boiler plate code)

mips32r2_uclibc_gclibcxx_dyn target compiles and tested to run on
production targets such as avm routers modified with a freetz env.

See cm8/freetz@41d97c3 for one of
many possible projects to build a working toolchain with.  In short
- git clone
- umask 0022
- make menuconfig (choose expert, disable toolchain download and
let the toolchain/make scripts built a gcc-5.x one, do not forget
to set FREETZ_LIB_libuClibc__WITH_WCHAR=y)
- read the commit message for further info on long double math
peculiarities
- tested here with 0.9.33.2

Remember that swap will need to be running on the box, or else
kiwix-serve is likely to quit with "invalid lzma stream in cluster"
errors (if memory is too low).

If the box lacks library support such as libstdc++.so*, you can
copy them from the target toolchain libdir over to
BUILD_mips32r2_uclibc_gclibcxx_dyn/INSTALL/lib if there are
unsatisfied dependencies at runtime.  (Which libraries need to be
supplemented this way depends on your firmware and/or freetz
configuration).

Another issue is the execution in non-standard installation
directories.  LD_LIBRARY_PATH env needs to point to "our" lib
directory.  If you tar INSTALL/ dir, transport the result to
an embedded device, untar and wan't to run from there, it is
best to wrap all the elf binaries with a shell script that
correctly sets LD_LIBRARY_PATH.  This step has been automated,
but needs testing, see
kiwixbuild/patches/fixenv-run-in-nonstd-installdir.sh
for details.

This fixenv script is copied to INSTALL/bin during the
build and it should be run on the box, if kiwix-tools is to
reside in nonstd location (i.e. if files are not installed
or installable to /bin, /lib or their usr/ pendants).

Feel free to improve on automation of the necessary setup
steps to make mips port build and deployment easier.
The following applies to mips32r2_uclibc_glibcxx_dyn target,
only:
  - libstdc++.so* tc libs are copied to INSTALL/lib/<full-arch>

  - i.e. there is an example showing how to copy other libs from
    the toolchain to supplement the installation files (in case
    they are found to be missing on a target machine)

  - if target already supplies libstdc++.so*, copied ones will be
    preferred for kiwix-tools binaries (when run from a non-std
    installation directory on the target), drawback in this case
    is extra space occupied by the lib, but the gain is less
    hazzle on target boxes that lack libstdc++.so*

  - comment or modify the lines in mips32r2.py accordingly to the
    setup of your mips target
necessary to output run-nonroot script correctly
the script won't create another file (run-nonroot previously)
and instead carry out environment setup itself for the binary
links pointing at it - it's more readable and smaller this way

rework mips32r2.py file copy def - may be useful in other files,
eventually transport this functionality to base.py (!?) later
on, after this branch is merged to master
@kelson42
Copy link
Contributor

@cm8 Thank you very much. What is the output of this? kiwix-tools for mips32r2 statically compiled? In addition, please rebase your branch.

@cm8
Copy link
Author

cm8 commented Jun 24, 2018

depends on the value used for --target-platform, as you can deduct from the PR comment, you can use it to build kiwix-tools dynamically and use that on a mips target router (even if you're not allowed to install kiwix binaries and libs in the standard locations on that device). Thus the compilation result in INSTALL may be used for two scenarios:

  • integrate them to the root directory structure of the target device, copying suffices (the usual way)
  • use them from a writable mount-point if root directory of the target is read-only/not-modifiable, copy and then call fixenv-nonstd-libdir on the target (kiwix-tools will be run through symlinks calling a shell script that sets up LD_LIBRARY_PATH in this case, for the linker to find the dynamic libraries in a non standard install location)

It may sound complicated, but use is straight-forward and its tried and tested to work.

@cm8
Copy link
Author

cm8 commented Jun 24, 2018

@kelson42 Can you include the target mips32r2_glibc_glibcxx_dyn in travis ci builds once it is merged to master? It should be possible, because the toolchain used for that target is in debian and thus apt-get-able. I'm not to experienced with editing the .travis.yml

as a side-note: it's not necessary to rebase this branch before its merged to master (it does not have conflicts with the 3 commits in between, so it can be merged at the tip of master right away)

@kelson42
Copy link
Contributor

@cm8 I will let @mgautierfr make the reviewing work. We definitely wants to (1) integrate the mips architecture to the CI (2) generate statically compiled nightlies/releases for mips. If all of this works fine we should then have a look how to make it as easy as possible for openWRT users and probably optimise a few things on kiwix-serve.... But before all of this happen, please do not forget to rebase your PR branch on master.

@stale
Copy link

stale bot commented Dec 14, 2019

This pull request has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

@stale stale bot added the stale label Dec 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants