New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port to Linux on ARM #61

Closed
ksbhaskar opened this Issue Oct 23, 2017 · 11 comments

Comments

Projects
None yet
5 participants
@ksbhaskar
Collaborator

ksbhaskar commented Oct 23, 2017

Final Release Note

YottaDB runs on Linux on ARM. Although the Supported platforms are Raspbian GNU/Linux 9.1 on Raspberry PI 3 Model B and Stretch IoT (non GUI) on BeagleBone Black Wireless, YottaDB should run on any contemporary Linux distribution on any CPU with an architecture upward compatible with ARMv7-A. Owing to variations in the implementations of ARM microarchitectures, we recommend that you ensure the software runs correctly before committing to any specific hardware other than those Supported. Please contact info@yottadb.com if you want a specific combination of OS and CPU microarchitecture to be Supported.

Language functionality, including call-in from and call-out to C, is identical to Linux on x86.

We recommend against attempting Asynchronous database IO (AIO) on this platform as the poor performance of AIO on our hardware precludes any meaningful testing of this functionality.

We would like to acknowledge and thank Steve Johnson (@sljohnson1) and Sam Habiel (@shabiel) for making the ARM port possible.

(YDB#61)

Description

Hardware such as the Raspberry Pi 3 provides a very cost effective, low power computing platform. YottaDB on that architecture opens up new applications.

Draft Release Note

YottaDB runs on Linux on ARM. Although the Supported platforms are Raspbian GNU/Linux 9.1 on Raspberry PI 3 Model B and Stretch IoT (non GUI) on BeagleBone Black Wireless, YottaDB should run on any contemporary Linux distribution on any CPU with an architecture upward compatible with ARMv7-A. We recommend against attempting Asynchronous database IO (AIO) on this platform as the poor performance of AIO on our hardware precludes any meaningful testing of this functionality.

@ksbhaskar ksbhaskar added this to the r110 milestone Oct 23, 2017

@shabiel

This comment has been minimized.

Show comment
Hide comment
@shabiel

shabiel Oct 24, 2017

Contributor

Yuppie!

Contributor

shabiel commented Oct 24, 2017

Yuppie!

@nars1 nars1 closed this Oct 26, 2017

nars1 added a commit to nars1/YottaDB that referenced this issue Jan 24, 2018

Merge GT.M V6.3-003 onto YottaDB mainline
Files that had merge conflicts which were manually resolved
-----------------------------------------------------------
* CMakeLists.txt
* sr_linux/platform.cmake
* sr_linux/release_name.h
* sr_port/bx_boollit.c
* sr_port/gdsfhead.h
* sr_port/gtm_env_init.c
* sr_port/mdb_condition_handler.c
* sr_port/mdef.h
* sr_port/merrors.msg
* sr_port/mumps.hlp
* sr_port/objlabel.h
* sr_port/op_bindparm.c
* sr_port/op_fnview.c
* sr_port/secshr_db_clnup.c
* sr_port/svnames.h
* sr_port_cm/gtcmd_cst_init.c
* sr_port_cm/gvcmz_lksublist.c
* sr_unix/gds_rundown.c
* sr_unix/jnlpool_init.c
* sr_unix_cm/omi_lkextnam.c
* sr_unix_cm/omi_prc_unlc.c
* sr_unix_cm/rc_prc_lock.c
* sr_port/parm_pool.h

* sr_port/op_halt.c
	--> Note: This module was deleted in GT.M V6.3-003

* sr_port/op_zhalt.c
	--> The returncode used to a pointer to an mval. Starting GT.M V6.3-003, this is an integer,
	--> so we need to do an i2mval() before setting TREF(gtmci_retval).

Files that had merge conflicts but the V63003 change was discarded
------------------------------------------------------------------
Reason for discard is mentioned below against each module.

* README
	--> Superseded in YottaDB by README.md

* sr_port/op_fnquery.c & sr_port/op_putindx.c
	--> GTM-6115/GTM-8792 in GT.M V6.3-003 release notes describes that only $QUERY
	--> on lvns with subscripts exceeding 1Mb in total length will be prohibited, not
	--> other operations like SET but the change in this module does the exact opposite.

* sr_port/stpg_sort.c
	--> This module was already changed in YottaDB (YottaDB#85) to significantly better stpg_sort performance.
	--> The V63003 change to this module seems to be a minor one and will take some time to
	--> see if can be retrofitted without losing the pre-existing YottaDB change so decided to discard.

* sr_unix/add_inter.c
	--> This module was already changed in YottaDB (YottaDB#61) to rework COMPSWAP_LOCK/COMPSWAP_UNLOCK macros.
	--> The V63003 change affected platforms that are not supported in YottaDB.

* sr_unix/configure.gtc
	--> The V63003 change to this module was already done in YottaDB as part of r1.10.

* sr_unix/err_init.c
	--> The V63003 change to this module was already done in YottaDB (Nov 2017).

* sr_unix/gtminstall.sh
	--> As part of r1.10 this module no longer exists (moved to ydbinstall.sh) and all changes
	--> done in V63003 seem to be already done or not needed for YottaDB.

* sr_unix/mupip_size.c
	--> This was already fixed differently (by removing the memset with the incorrect size) in YottaDB.
	--> The V63003 change fixed it by doing the memset with the correct size.

* sr_x86_64/merrors_ansi.h
* sr_x86_64/merrors_ctl.c
* sr_i386/gdeerrors_ctl.c
	--> All these modules have been moved to sr_port in YottaDB (thereby avoiding duplication across
	--> multiple supported platform directories).

Files that had merge conflicts which were resolved by regenerating them
-----------------------------------------------------------------------
* sr_port/merrors_ctl.c
* sr_port/merrors_ansi.h
* sr_x86_64/ttt.c
* sr_armv7l/ttt.c
* sr_x86_64/GTMDefinedTypesInitDebug.m
* sr_x86_64/GTMDefinedTypesInitRelease.m
* sr_armv7l/GTMDefinedTypesInitDebug.m
* sr_armv7l/GTMDefinedTypesInitRelease.m

Files that did not have any merge conflicts between YottaDB mainline and GT.M V6.3-003
--------------------------------------------------------------------------------------
This list includes all other files (not mentioned above) and part of this commit.
All these files were merged as is from V63003 (i.e. had no merge conflicts).

nars1 added a commit that referenced this issue Jan 25, 2018

Merge GT.M V6.3-003 onto YottaDB mainline
Files that had merge conflicts which were manually resolved
-----------------------------------------------------------
* CMakeLists.txt
* sr_linux/platform.cmake
* sr_linux/release_name.h
* sr_port/bx_boollit.c
* sr_port/gdsfhead.h
* sr_port/gtm_env_init.c
* sr_port/mdb_condition_handler.c
* sr_port/mdef.h
* sr_port/merrors.msg
* sr_port/mumps.hlp
* sr_port/objlabel.h
* sr_port/op_bindparm.c
* sr_port/op_fnview.c
* sr_port/secshr_db_clnup.c
* sr_port/svnames.h
* sr_port_cm/gtcmd_cst_init.c
* sr_port_cm/gvcmz_lksublist.c
* sr_unix/gds_rundown.c
* sr_unix/jnlpool_init.c
* sr_unix_cm/omi_lkextnam.c
* sr_unix_cm/omi_prc_unlc.c
* sr_unix_cm/rc_prc_lock.c
* sr_port/parm_pool.h

* sr_port/op_halt.c
	--> Note: This module was deleted in GT.M V6.3-003

* sr_port/op_zhalt.c
	--> The returncode used to a pointer to an mval. Starting GT.M V6.3-003, this is an integer,
	--> so we need to do an i2mval() before setting TREF(gtmci_retval).

Files that had merge conflicts but the V63003 change was discarded
------------------------------------------------------------------
Reason for discard is mentioned below against each module.

* README
	--> Superseded in YottaDB by README.md

* sr_port/op_fnquery.c & sr_port/op_putindx.c
	--> GTM-6115/GTM-8792 in GT.M V6.3-003 release notes describes that only $QUERY
	--> on lvns with subscripts exceeding 1Mb in total length will be prohibited, not
	--> other operations like SET but the change in this module does the exact opposite.

* sr_port/stpg_sort.c
	--> This module was already changed in YottaDB (#85) to significantly better stpg_sort performance.
	--> The V63003 change to this module seems to be a minor one and will take some time to
	--> see if can be retrofitted without losing the pre-existing YottaDB change so decided to discard.

* sr_unix/add_inter.c
	--> This module was already changed in YottaDB (#61) to rework COMPSWAP_LOCK/COMPSWAP_UNLOCK macros.
	--> The V63003 change affected platforms that are not supported in YottaDB.

* sr_unix/configure.gtc
	--> The V63003 change to this module was already done in YottaDB as part of r1.10.

* sr_unix/err_init.c
	--> The V63003 change to this module was already done in YottaDB (Nov 2017).

* sr_unix/gtminstall.sh
	--> As part of r1.10 this module no longer exists (moved to ydbinstall.sh) and all changes
	--> done in V63003 seem to be already done or not needed for YottaDB.

* sr_unix/mupip_size.c
	--> This was already fixed differently (by removing the memset with the incorrect size) in YottaDB.
	--> The V63003 change fixed it by doing the memset with the correct size.

* sr_x86_64/merrors_ansi.h
* sr_x86_64/merrors_ctl.c
* sr_i386/gdeerrors_ctl.c
	--> All these modules have been moved to sr_port in YottaDB (thereby avoiding duplication across
	--> multiple supported platform directories).

Files that had merge conflicts which were resolved by regenerating them
-----------------------------------------------------------------------
* sr_port/merrors_ctl.c
* sr_port/merrors_ansi.h
* sr_x86_64/ttt.c
* sr_armv7l/ttt.c
* sr_x86_64/GTMDefinedTypesInitDebug.m
* sr_x86_64/GTMDefinedTypesInitRelease.m
* sr_armv7l/GTMDefinedTypesInitDebug.m
* sr_armv7l/GTMDefinedTypesInitRelease.m

Files that did not have any merge conflicts between YottaDB mainline and GT.M V6.3-003
--------------------------------------------------------------------------------------
This list includes all other files (not mentioned above) and part of this commit.
All these files were merged as is from V63003 (i.e. had no merge conflicts).
@yaweli

This comment has been minimized.

Show comment
Hide comment
@yaweli

yaweli Jul 7, 2018

Hi
try to install on pi 2,
script run and complete with:
YottaDB version r1.22 installed successfully at /usr/local/lib/yottadb/r122
but nothing there , directory no exist.

tyrinstallpi.txt

thank you
eli

yaweli commented Jul 7, 2018

Hi
try to install on pi 2,
script run and complete with:
YottaDB version r1.22 installed successfully at /usr/local/lib/yottadb/r122
but nothing there , directory no exist.

tyrinstallpi.txt

thank you
eli

@nars1

This comment has been minimized.

Show comment
Hide comment
@nars1

nars1 Jul 7, 2018

Collaborator

@yaweli : We don't have a Pi 2 box in-house so are not sure how the script would behave there. A few questions.

  1. Where did you get the ydbinstall.sh script from? Can you send us a "ls -l" output of that script.
  2. Can you send us a "uname -a" output on your system.
Collaborator

nars1 commented Jul 7, 2018

@yaweli : We don't have a Pi 2 box in-house so are not sure how the script would behave there. A few questions.

  1. Where did you get the ydbinstall.sh script from? Can you send us a "ls -l" output of that script.
  2. Can you send us a "uname -a" output on your system.
@yaweli

This comment has been minimized.

Show comment
Hide comment
@yaweli

yaweli Jul 7, 2018

yaweli commented Jul 7, 2018

@nars1

This comment has been minimized.

Show comment
Hide comment
@nars1

nars1 Jul 7, 2018

Collaborator

Can you use the following ydbinstall.sh instead.

https://github.com/nars1/YottaDB/blob/ydbinstall/sr_unix/ydbinstall.sh

Also, redirect the output of the script to a file, say install.out

$ sh -x ./ydbinstall.sh > install.out

And then determine the temporary directory name as follows.

$ grep "tmpdirname=" install.out
+ tmpdirname=/tmp/tmp.PwrToeBlhS

And then list the contents of that temporary directory. You should see output like the following.

$ sudo ls -l /tmp/tmp.PwrToeBlhS
total 3644
-rw-r--r-- 1 root root   57145 Jul  7 14:32 configure_20180707143119.err
-rw-r--r-- 1 root root      50 Jul  7 14:31 configure_20180707143119.in
-rw-r--r-- 1 root root    2677 Jul  7 14:32 configure_20180707143119.out
-rw-r--r-- 1 root root   64510 Jul  2 13:25 latest
-rw-r--r-- 1 root root   64510 Jul  2 13:25 r1.22
-rw-r--r-- 1 root root       0 Jul  7 14:31 tar.log
drwxr-xr-x 3 root root    4096 Jul  7 14:31 tmp
-rw-r--r-- 1 root root       0 Jul  7 14:31 wget_dist.log
-rw-r--r-- 1 root root       0 Jul  7 14:31 wget_latest.log
-rw-r--r-- 1 root root 3530055 May 16 16:21 yottadb_r122_linux_armv6l_pro.tgz

Please upload (do not copy/paste) the configure_* files (.in, .out and .err). That might help understand what is going on.

Also, in the future, please open a new issue for this installation problem. Thanks.

Collaborator

nars1 commented Jul 7, 2018

Can you use the following ydbinstall.sh instead.

https://github.com/nars1/YottaDB/blob/ydbinstall/sr_unix/ydbinstall.sh

Also, redirect the output of the script to a file, say install.out

$ sh -x ./ydbinstall.sh > install.out

And then determine the temporary directory name as follows.

$ grep "tmpdirname=" install.out
+ tmpdirname=/tmp/tmp.PwrToeBlhS

And then list the contents of that temporary directory. You should see output like the following.

$ sudo ls -l /tmp/tmp.PwrToeBlhS
total 3644
-rw-r--r-- 1 root root   57145 Jul  7 14:32 configure_20180707143119.err
-rw-r--r-- 1 root root      50 Jul  7 14:31 configure_20180707143119.in
-rw-r--r-- 1 root root    2677 Jul  7 14:32 configure_20180707143119.out
-rw-r--r-- 1 root root   64510 Jul  2 13:25 latest
-rw-r--r-- 1 root root   64510 Jul  2 13:25 r1.22
-rw-r--r-- 1 root root       0 Jul  7 14:31 tar.log
drwxr-xr-x 3 root root    4096 Jul  7 14:31 tmp
-rw-r--r-- 1 root root       0 Jul  7 14:31 wget_dist.log
-rw-r--r-- 1 root root       0 Jul  7 14:31 wget_latest.log
-rw-r--r-- 1 root root 3530055 May 16 16:21 yottadb_r122_linux_armv6l_pro.tgz

Please upload (do not copy/paste) the configure_* files (.in, .out and .err). That might help understand what is going on.

Also, in the future, please open a new issue for this installation problem. Thanks.

@yaweli

This comment has been minimized.

Show comment
Hide comment
@yaweli

yaweli Jul 28, 2018

root@pi2:/home/pi/dev/mumps/yot# rm -fr *
root@pi2:/home/pi/dev/mumps/yot# wget https://raw.githubusercontent.com/nars1/YottaDB/ydbinstall/sr_unix/ydbinstall.sh
--2018-07-28 12:33:50-- https://raw.githubusercontent.com/nars1/YottaDB/ydbinstall/sr_unix/ydbinstall.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.112.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.112.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30495 (30K) [text/plain]
Saving to: `ydbinstall.sh'

100%[============================================================================================================================================>] 30,495 --.-K/s in 0.09s

2018-07-28 12:33:56 (320 KB/s) - `ydbinstall.sh' saved [30495/30495]

root@pi2:/home/pi/dev/mumps/yot# sh -x ./ydbinstall.sh > install.out 2>yerr.out
root@pi2:/home/pi/dev/mumps/yot# grep "tmpdirname=" *.out
yerr.out:+ tmpdirname=/tmp/tmp.QTcHg6Q4co
root@pi2:/home/pi/dev/mumps/yot# ls -l /tmp/tmp.QTcHg6Q4co
total 3592
-rw-r--r-- 1 root root 3013 Jul 28 12:34 configure_20180728123416.err
-rw-r--r-- 1 root root 50 Jul 28 12:34 configure_20180728123416.in
-rw-r--r-- 1 root root 32 Jul 28 12:34 configure_20180728123416.out
-rw-r--r-- 1 root root 63932 Jul 10 20:49 latest
-rw-r--r-- 1 root root 63932 Jul 10 20:49 r1.22
-rw-r--r-- 1 root root 0 Jul 28 12:34 tar.log
drwxr-xr-x 3 root root 4096 Jul 28 12:34 tmp
-rw-r--r-- 1 root root 0 Jul 28 12:34 wget_dist.log
-rw-r--r-- 1 root root 0 Jul 28 12:34 wget_latest.log
-rw-r--r-- 1 root root 3530055 May 16 23:21 yottadb_r122_linux_armv6l_pro.tgz
root@pi2:/home/pi/dev/mumps/yot#

(i have renamed the files all to *.txt , so git website will agree to upload them)

configure_20180728123416.err.txt
configure_20180728123416.in.txt
configure_20180728123416.out.txt
install.out.txt
yerr.out.txt

Thank you
Eli

yaweli commented Jul 28, 2018

root@pi2:/home/pi/dev/mumps/yot# rm -fr *
root@pi2:/home/pi/dev/mumps/yot# wget https://raw.githubusercontent.com/nars1/YottaDB/ydbinstall/sr_unix/ydbinstall.sh
--2018-07-28 12:33:50-- https://raw.githubusercontent.com/nars1/YottaDB/ydbinstall/sr_unix/ydbinstall.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.112.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.112.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30495 (30K) [text/plain]
Saving to: `ydbinstall.sh'

100%[============================================================================================================================================>] 30,495 --.-K/s in 0.09s

2018-07-28 12:33:56 (320 KB/s) - `ydbinstall.sh' saved [30495/30495]

root@pi2:/home/pi/dev/mumps/yot# sh -x ./ydbinstall.sh > install.out 2>yerr.out
root@pi2:/home/pi/dev/mumps/yot# grep "tmpdirname=" *.out
yerr.out:+ tmpdirname=/tmp/tmp.QTcHg6Q4co
root@pi2:/home/pi/dev/mumps/yot# ls -l /tmp/tmp.QTcHg6Q4co
total 3592
-rw-r--r-- 1 root root 3013 Jul 28 12:34 configure_20180728123416.err
-rw-r--r-- 1 root root 50 Jul 28 12:34 configure_20180728123416.in
-rw-r--r-- 1 root root 32 Jul 28 12:34 configure_20180728123416.out
-rw-r--r-- 1 root root 63932 Jul 10 20:49 latest
-rw-r--r-- 1 root root 63932 Jul 10 20:49 r1.22
-rw-r--r-- 1 root root 0 Jul 28 12:34 tar.log
drwxr-xr-x 3 root root 4096 Jul 28 12:34 tmp
-rw-r--r-- 1 root root 0 Jul 28 12:34 wget_dist.log
-rw-r--r-- 1 root root 0 Jul 28 12:34 wget_latest.log
-rw-r--r-- 1 root root 3530055 May 16 23:21 yottadb_r122_linux_armv6l_pro.tgz
root@pi2:/home/pi/dev/mumps/yot#

(i have renamed the files all to *.txt , so git website will agree to upload them)

configure_20180728123416.err.txt
configure_20180728123416.in.txt
configure_20180728123416.out.txt
install.out.txt
yerr.out.txt

Thank you
Eli

@nars1

This comment has been minimized.

Show comment
Hide comment
@nars1

nars1 Aug 2, 2018

Collaborator

Thanks. The output file shows the issue.

$ cat configure_20180728123416.out.txt
You must run Configure as root.

You need to run ydbinstall as root. Let us know how that goes. We have made a note to enhance the ydbinstall script to indicate failure if the configure script it invokes internally fails.

Collaborator

nars1 commented Aug 2, 2018

Thanks. The output file shows the issue.

$ cat configure_20180728123416.out.txt
You must run Configure as root.

You need to run ydbinstall as root. Let us know how that goes. We have made a note to enhance the ydbinstall script to indicate failure if the configure script it invokes internally fails.

@nars1

This comment has been minimized.

Show comment
Hide comment
@nars1

nars1 Aug 9, 2018

Collaborator

@yaweli : Ignore my previous comment. That error about "You must run Configure as root" is not the primary issue. One of the executables "geteuid" (which is used to determine if the effective user id is root or not) could not run because of the below error (in line 47).

configure_20180728123416.err.txt
46 + ./geteuid
47 ./geteuid: error while loading shared libraries: libelf.so.1: cannot open shared object file: No such file or directory

And that caused the configure script to incorrectly think the caller is not root.

Nevertheless, the error in line 47 above indicates to me that it is not straightforward to get YottaDB running on your Pi 2 system. And since we do not have a Pi 2 system, we will not be able to spend a lot more time on this one. Hope you understand. Thanks.

Collaborator

nars1 commented Aug 9, 2018

@yaweli : Ignore my previous comment. That error about "You must run Configure as root" is not the primary issue. One of the executables "geteuid" (which is used to determine if the effective user id is root or not) could not run because of the below error (in line 47).

configure_20180728123416.err.txt
46 + ./geteuid
47 ./geteuid: error while loading shared libraries: libelf.so.1: cannot open shared object file: No such file or directory

And that caused the configure script to incorrectly think the caller is not root.

Nevertheless, the error in line 47 above indicates to me that it is not straightforward to get YottaDB running on your Pi 2 system. And since we do not have a Pi 2 system, we will not be able to spend a lot more time on this one. Hope you understand. Thanks.

@ChristopherEdwards

This comment has been minimized.

Show comment
Hide comment
@ChristopherEdwards

ChristopherEdwards Aug 9, 2018

Member

@yaweli

You may want to try installing the libelf library: sudo apt-get install libelf1

Member

ChristopherEdwards commented Aug 9, 2018

@yaweli

You may want to try installing the libelf library: sudo apt-get install libelf1

@nars1

This comment has been minimized.

Show comment
Hide comment
@nars1

nars1 Aug 9, 2018

Collaborator
Collaborator

nars1 commented Aug 9, 2018

@nars1

This comment has been minimized.

Show comment
Hide comment
@nars1

nars1 Aug 9, 2018

Collaborator

@yaweli : FYI. This note is regarding my previous comment (We have made a note to enhance the ydbinstall script to indicate failure if the configure script it invokes internally fails). This is now fixed so the ydbinstall.sh script will no longer incorrectly signal a success if the configure script (that it internally invokes) fails. The link to that issue is #338.

Collaborator

nars1 commented Aug 9, 2018

@yaweli : FYI. This note is regarding my previous comment (We have made a note to enhance the ydbinstall script to indicate failure if the configure script it invokes internally fails). This is now fixed so the ydbinstall.sh script will no longer incorrectly signal a success if the configure script (that it internally invokes) fails. The link to that issue is #338.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment