Port to Linux on ARM #61
Comments
|
Yuppie! |
Files that had merge conflicts which were manually resolved ----------------------------------------------------------- * CMakeLists.txt * sr_linux/platform.cmake * sr_linux/release_name.h * sr_port/bx_boollit.c * sr_port/gdsfhead.h * sr_port/gtm_env_init.c * sr_port/mdb_condition_handler.c * sr_port/mdef.h * sr_port/merrors.msg * sr_port/mumps.hlp * sr_port/objlabel.h * sr_port/op_bindparm.c * sr_port/op_fnview.c * sr_port/secshr_db_clnup.c * sr_port/svnames.h * sr_port_cm/gtcmd_cst_init.c * sr_port_cm/gvcmz_lksublist.c * sr_unix/gds_rundown.c * sr_unix/jnlpool_init.c * sr_unix_cm/omi_lkextnam.c * sr_unix_cm/omi_prc_unlc.c * sr_unix_cm/rc_prc_lock.c * sr_port/parm_pool.h * sr_port/op_halt.c --> Note: This module was deleted in GT.M V6.3-003 * sr_port/op_zhalt.c --> The returncode used to a pointer to an mval. Starting GT.M V6.3-003, this is an integer, --> so we need to do an i2mval() before setting TREF(gtmci_retval). Files that had merge conflicts but the V63003 change was discarded ------------------------------------------------------------------ Reason for discard is mentioned below against each module. * README --> Superseded in YottaDB by README.md * sr_port/op_fnquery.c & sr_port/op_putindx.c --> GTM-6115/GTM-8792 in GT.M V6.3-003 release notes describes that only $QUERY --> on lvns with subscripts exceeding 1Mb in total length will be prohibited, not --> other operations like SET but the change in this module does the exact opposite. * sr_port/stpg_sort.c --> This module was already changed in YottaDB (YottaDB#85) to significantly better stpg_sort performance. --> The V63003 change to this module seems to be a minor one and will take some time to --> see if can be retrofitted without losing the pre-existing YottaDB change so decided to discard. * sr_unix/add_inter.c --> This module was already changed in YottaDB (YottaDB#61) to rework COMPSWAP_LOCK/COMPSWAP_UNLOCK macros. --> The V63003 change affected platforms that are not supported in YottaDB. * sr_unix/configure.gtc --> The V63003 change to this module was already done in YottaDB as part of r1.10. * sr_unix/err_init.c --> The V63003 change to this module was already done in YottaDB (Nov 2017). * sr_unix/gtminstall.sh --> As part of r1.10 this module no longer exists (moved to ydbinstall.sh) and all changes --> done in V63003 seem to be already done or not needed for YottaDB. * sr_unix/mupip_size.c --> This was already fixed differently (by removing the memset with the incorrect size) in YottaDB. --> The V63003 change fixed it by doing the memset with the correct size. * sr_x86_64/merrors_ansi.h * sr_x86_64/merrors_ctl.c * sr_i386/gdeerrors_ctl.c --> All these modules have been moved to sr_port in YottaDB (thereby avoiding duplication across --> multiple supported platform directories). Files that had merge conflicts which were resolved by regenerating them ----------------------------------------------------------------------- * sr_port/merrors_ctl.c * sr_port/merrors_ansi.h * sr_x86_64/ttt.c * sr_armv7l/ttt.c * sr_x86_64/GTMDefinedTypesInitDebug.m * sr_x86_64/GTMDefinedTypesInitRelease.m * sr_armv7l/GTMDefinedTypesInitDebug.m * sr_armv7l/GTMDefinedTypesInitRelease.m Files that did not have any merge conflicts between YottaDB mainline and GT.M V6.3-003 -------------------------------------------------------------------------------------- This list includes all other files (not mentioned above) and part of this commit. All these files were merged as is from V63003 (i.e. had no merge conflicts).
Files that had merge conflicts which were manually resolved ----------------------------------------------------------- * CMakeLists.txt * sr_linux/platform.cmake * sr_linux/release_name.h * sr_port/bx_boollit.c * sr_port/gdsfhead.h * sr_port/gtm_env_init.c * sr_port/mdb_condition_handler.c * sr_port/mdef.h * sr_port/merrors.msg * sr_port/mumps.hlp * sr_port/objlabel.h * sr_port/op_bindparm.c * sr_port/op_fnview.c * sr_port/secshr_db_clnup.c * sr_port/svnames.h * sr_port_cm/gtcmd_cst_init.c * sr_port_cm/gvcmz_lksublist.c * sr_unix/gds_rundown.c * sr_unix/jnlpool_init.c * sr_unix_cm/omi_lkextnam.c * sr_unix_cm/omi_prc_unlc.c * sr_unix_cm/rc_prc_lock.c * sr_port/parm_pool.h * sr_port/op_halt.c --> Note: This module was deleted in GT.M V6.3-003 * sr_port/op_zhalt.c --> The returncode used to a pointer to an mval. Starting GT.M V6.3-003, this is an integer, --> so we need to do an i2mval() before setting TREF(gtmci_retval). Files that had merge conflicts but the V63003 change was discarded ------------------------------------------------------------------ Reason for discard is mentioned below against each module. * README --> Superseded in YottaDB by README.md * sr_port/op_fnquery.c & sr_port/op_putindx.c --> GTM-6115/GTM-8792 in GT.M V6.3-003 release notes describes that only $QUERY --> on lvns with subscripts exceeding 1Mb in total length will be prohibited, not --> other operations like SET but the change in this module does the exact opposite. * sr_port/stpg_sort.c --> This module was already changed in YottaDB (#85) to significantly better stpg_sort performance. --> The V63003 change to this module seems to be a minor one and will take some time to --> see if can be retrofitted without losing the pre-existing YottaDB change so decided to discard. * sr_unix/add_inter.c --> This module was already changed in YottaDB (#61) to rework COMPSWAP_LOCK/COMPSWAP_UNLOCK macros. --> The V63003 change affected platforms that are not supported in YottaDB. * sr_unix/configure.gtc --> The V63003 change to this module was already done in YottaDB as part of r1.10. * sr_unix/err_init.c --> The V63003 change to this module was already done in YottaDB (Nov 2017). * sr_unix/gtminstall.sh --> As part of r1.10 this module no longer exists (moved to ydbinstall.sh) and all changes --> done in V63003 seem to be already done or not needed for YottaDB. * sr_unix/mupip_size.c --> This was already fixed differently (by removing the memset with the incorrect size) in YottaDB. --> The V63003 change fixed it by doing the memset with the correct size. * sr_x86_64/merrors_ansi.h * sr_x86_64/merrors_ctl.c * sr_i386/gdeerrors_ctl.c --> All these modules have been moved to sr_port in YottaDB (thereby avoiding duplication across --> multiple supported platform directories). Files that had merge conflicts which were resolved by regenerating them ----------------------------------------------------------------------- * sr_port/merrors_ctl.c * sr_port/merrors_ansi.h * sr_x86_64/ttt.c * sr_armv7l/ttt.c * sr_x86_64/GTMDefinedTypesInitDebug.m * sr_x86_64/GTMDefinedTypesInitRelease.m * sr_armv7l/GTMDefinedTypesInitDebug.m * sr_armv7l/GTMDefinedTypesInitRelease.m Files that did not have any merge conflicts between YottaDB mainline and GT.M V6.3-003 -------------------------------------------------------------------------------------- This list includes all other files (not mentioned above) and part of this commit. All these files were merged as is from V63003 (i.e. had no merge conflicts).
|
Hi thank you |
|
@yaweli : We don't have a Pi 2 box in-house so are not sure how the script would behave there. A few questions.
|
|
https://github.com/YottaDB/YottaDB/blob/master/sr_unix/ydbinstall.sh
root@pi2:/home/pi/dev/mumps/yot# uname -a
Linux pi2 3.18.11+ #781 PREEMPT Tue Apr 21 18:02:18 BST 2015 armv6l
GNU/Linux
root@pi2:/home/pi/dev/mumps/yot#
Thank you
*Eli Smadar*
|
|
Can you use the following ydbinstall.sh instead. https://github.com/nars1/YottaDB/blob/ydbinstall/sr_unix/ydbinstall.sh Also, redirect the output of the script to a file, say install.out And then determine the temporary directory name as follows. And then list the contents of that temporary directory. You should see output like the following. Please upload (do not copy/paste) the configure_* files (.in, .out and .err). That might help understand what is going on. Also, in the future, please open a new issue for this installation problem. Thanks. |
|
root@pi2:/home/pi/dev/mumps/yot# rm -fr * 100%[============================================================================================================================================>] 30,495 --.-K/s in 0.09s 2018-07-28 12:33:56 (320 KB/s) - `ydbinstall.sh' saved [30495/30495] root@pi2:/home/pi/dev/mumps/yot# sh -x ./ydbinstall.sh > install.out 2>yerr.out (i have renamed the files all to *.txt , so git website will agree to upload them) configure_20180728123416.err.txt Thank you |
|
Thanks. The output file shows the issue. $ cat configure_20180728123416.out.txt You need to run ydbinstall as root. Let us know how that goes. We have made a note to enhance the ydbinstall script to indicate failure if the configure script it invokes internally fails. |
|
@yaweli : Ignore my previous comment. That error about "You must run Configure as root" is not the primary issue. One of the executables "geteuid" (which is used to determine if the effective user id is root or not) could not run because of the below error (in line 47). configure_20180728123416.err.txt And that caused the configure script to incorrectly think the caller is not root. Nevertheless, the error in line 47 above indicates to me that it is not straightforward to get YottaDB running on your Pi 2 system. And since we do not have a Pi 2 system, we will not be able to spend a lot more time on this one. Hope you understand. Thanks. |
|
You may want to try installing the libelf library: |
|
Thanks @ChristopherEdwards. |
|
@yaweli : FYI. This note is regarding my previous comment (We have made a note to enhance the ydbinstall script to indicate failure if the configure script it invokes internally fails). This is now fixed so the ydbinstall.sh script will no longer incorrectly signal a success if the configure script (that it internally invokes) fails. The link to that issue is #338. |
* The below M program spawns off around 1000 child processes each of which immediately quit.
This uses `com/job.m` from the YDBTest repo which does M lock operations in the parent
and child processes.
```m
$ cat x.m
x ;
do ^job("child^x",500+$random(1000),"""""")
quit
child ;
quit
```
When running this M program around a 100 times or so, some of the child processes were found
to terminate abnormally with an assert failure around 5% of the test runs.
```
%YDB-F-ASSERT, Assert failed in sr_port/mlk_garbage_collect.h line 59 for expression ((&ctl->lock_gc_in_progress)->u.parts.latch_pid == old_gc)
```
With the following C-stack trace.
```c
(gdb) where
#0 pthread_kill () from /usr/lib/libpthread.so.0
#1 gtm_dump_core () at sr_unix/gtm_dump_core.c:74
#2 gtm_fork_n_core () at sr_unix/gtm_fork_n_core.c:163
#3 ch_cond_core () at sr_unix/ch_cond_core.c:80
#4 rts_error_va (csa=0x0, argcnt=7, var=0x7ffed96b4e80) at sr_unix/rts_error.c:192
#5 rts_error_csa (csa=0x0, argcnt=7) at sr_unix/rts_error.c:99
#6 prepare_for_gc (pctl=0x5558188b1340) at sr_port/mlk_garbage_collect.h:59
#7 mlk_lock (p=0x5558188b1340, auxown=0, new=0 '\000') at sr_port/mlk_lock.c:114
#8 op_lock2_common (timeout=9223372036854775800, laflag=0 '\000') at sr_port/op_lock2.c:383
#9 op_lock2 (timeout=0x7fdc779f7070, laflag=0 '\000') at sr_port/op_lock2.c:158
#10 op_lock (timeout=0x7fdc779f7070) at sr_port/op_lock.c:36
(gdb) f 6
#6 0x00007f8ee778b1a8 in prepare_for_gc (pctl=0x55b034213c40) at /Distrib/YottaDB/V999_R129/sr_port/mlk_garbage_collect.h:59
59 COMPSWAP_UNLOCK(&ctl->lock_gc_in_progress, old_gc, LOCK_AVAILABLE);
(gdb) p ctl->lock_gc_in_progress->u.parts.latch_pid
0
(gdb) p old_gc
$2 = 37247
(gdb) p process_id
$3 = 37019
```
The assert failure is because the process in `prepare_for_gc()` P1 found a pid P2 to be holding
the latch and later found P2 exited. So P1 invoked the `COMPSWAP_UNLOCK` macro on the latch using
P1 as the old value of the latch. But it turns out that the latch had already been released.
This is because P2 had released it as part of its exit processing.
The assert in COMPSWAP_UNLOCK is incorrect in this case because this is a situation where the latch
holder is not releasing it but a completely unrelated process is trying to salvage the latch thinking
it is being held by a dead process.
This assert was added to the COMPSWAP_UNLOCK macro as part of #61 (commit @26391b73). While this
assert was true at that time, since then a few cases were added in concurrent GT.M releases
(e.g. `jnl_write_attempt()`, `prepare_for_gc()`) where lock salvaging code used COMPSWAP_UNLOCK.
In pro code, the COMSWAP_UNLOCK macro does the right thing by releasing the latch/lock only if CMPVAL
passed in is identical to the latch value.
Therefore there is no real issue. Since the assert is not accurate in rare cases, and since the pro
code already does the right thing, that portion of the assert is now removed.
With the changes, the test was run 100 times again and this time around there were no assert failures.
Final Release Note
YottaDB runs on Linux on ARM. Although the Supported platforms are Raspbian GNU/Linux 9.1 on Raspberry PI 3 Model B and Stretch IoT (non GUI) on BeagleBone Black Wireless, YottaDB should run on any contemporary Linux distribution on any CPU with an architecture upward compatible with ARMv7-A. Owing to variations in the implementations of ARM microarchitectures, we recommend that you ensure the software runs correctly before committing to any specific hardware other than those Supported. Please contact info@yottadb.com if you want a specific combination of OS and CPU microarchitecture to be Supported.
Language functionality, including call-in from and call-out to C, is identical to Linux on x86.
We recommend against attempting Asynchronous database IO (AIO) on this platform as the poor performance of AIO on our hardware precludes any meaningful testing of this functionality.
We would like to acknowledge and thank Steve Johnson (@sljohnson1) and Sam Habiel (@shabiel) for making the ARM port possible.
(YDB#61)
Description
Hardware such as the Raspberry Pi 3 provides a very cost effective, low power computing platform. YottaDB on that architecture opens up new applications.
Draft Release Note
YottaDB runs on Linux on ARM. Although the Supported platforms are Raspbian GNU/Linux 9.1 on Raspberry PI 3 Model B and Stretch IoT (non GUI) on BeagleBone Black Wireless, YottaDB should run on any contemporary Linux distribution on any CPU with an architecture upward compatible with ARMv7-A. We recommend against attempting Asynchronous database IO (AIO) on this platform as the poor performance of AIO on our hardware precludes any meaningful testing of this functionality.
The text was updated successfully, but these errors were encountered: