Running WebODM on Apple Silicon M1 fails

[p-vitt]

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

Firefox 89 on macOS (Catalina)

What is the problem?

When running WebODM, the db container keeps restarting. It keeps throwing errors like this:

db | runtime: failed to create new OS thread (have 2 already; errno=22)
db | fatal error: newosproc
db |
db | runtime stack:
db | runtime.throw(0x524da0, 0x9)
db | /usr/local/go/src/runtime/panic.go:527 +0x90
db | runtime.newosproc(0xc820028000, 0xc820037fc0)
db | /usr/local/go/src/runtime/os1_linux.go:150 +0x1ab
db | runtime.newm(0x555ce8, 0x0)
db | /usr/local/go/src/runtime/proc1.go:1105 +0x130
db | runtime.main.func1()
db | /usr/local/go/src/runtime/proc.go:48 +0x2c
db | runtime.systemstack(0x5c4300)
db | /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
db | runtime.mstart()
db | /usr/local/go/src/runtime/proc1.go:674
db |
db | goroutine 1 [running]:
db | runtime.systemstack_switch()
db | /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc820022770 sp=0xc820022768
db | runtime.main()
db | /usr/local/go/src/runtime/proc.go:49 +0x62 fp=0xc8200227c0 sp=0xc820022770
db | runtime.goexit()
db | /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200227c8 sp=0xc8200227c0
...
webapp | psql: error: could not connect to server: Connection refused
webapp | Is the server running on host "db" (172.21.0.2) and accepting
webapp | TCP/IP connections on port 5432?
webapp | Postgres is unavailable - sleeping
...
db exited with code 2

What should be the expected behavior?

The db container should run.

How can we reproduce this? (What steps did you do to trigger the problem? What parameters are you using for processing? Please include multiple screenshots of the problem! If possible please include a copy of your dataset uploaded on Google Drive or Dropbox. Be detailed)

I basically installed WebODM as described in the manual.

[p-vitt]

This problem seems to be related to the cpu architecture of the Apple Silicon M1. The go runtime tries to execute some amd64 instruction, which is not present on the arm architecture.

As only the database seems to be affected, my first try was to update this container to a newer version:

diff --git a/db/Dockerfile b/db/Dockerfile
index 47d7278..fe766f5 100644
--- a/db/Dockerfile
+++ b/db/Dockerfile
@@ -1,7 +1,7 @@
-FROM postgres:9.5
+FROM postgres:13
MAINTAINER Piero Toffanin pt@masseranolabs.com

-ENV POSTGIS_MAJOR 2.3
+ENV POSTGIS_MAJOR 3
ENV POSTGRES_PASSWORD postgres

RUN . /etc/os-release \

This container seems to run, however, I wasn't able to get all the other containers up and running. A ./webodm.sh start just ran the old database image, not the new one, so I did a ./webodm.sh rebuild, which started to also rebuild the working images, just to fail on them. I'm quite new to docker, thus it can be due to my lack of knowledge. But as the webodm image was failing, I tried to update it's Dockerfile in the main folder as well:

diff --git a/Dockerfile b/Dockerfile
index fcd21b3..45c33db 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -4,6 +4,7 @@ MAINTAINER Piero Toffanin pt@masseranolabs.com
ENV PYTHONUNBUFFERED 1
ENV PYTHONPATH $PYTHONPATH:/webodm
ENV PROJ_LIB=/usr/share/proj
+ENV GDAL_VERSION=3.3.0

RUN mkdir /webodm
@@ -18,7 +19,7 @@ RUN wget --no-check-certificate https://deb.nodesource.com/setup_12.x -O /tmp/no
RUN apt-get -qq update && apt-get -qq install -y nodejs

-RUN apt-get -qq update && apt-get -qq install -y --no-install-recommends python3 python3-pip python3-setuptools python3-wheel git g++ python3-dev python2.7-dev libpq-dev binutils libproj-dev gdal-bin python3-gdal nginx certbot grass-core gettext-base cron postgresql-client-12 gettext
+RUN apt-get -qq update && apt-get -qq install -y --no-install-recommends python3 python3-pip python3-setuptools python3-wheel git g++ python3-dev python2.7-dev libpq-dev binutils libproj-dev gdal-bin libgdal-dev python3-gdal nginx certbot grass-core gettext-base cron postgresql-client-12 gettext
RUN update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && update-alternatives --install /usr/bin/python python /usr/bin/python3.8 2

But this leaves me with an error in building the container:

24 0.867
24 0.867 During handling of the above exception, another exception occurred:
24 0.867
24 0.867 Traceback (most recent call last):
24 0.867 File "manage.py", line 22, in
24 0.867 execute_from_command_line(sys.argv)
24 0.867 File "/usr/local/lib/python3.8/dist-packages/django/core/management/init.py", line 419, in >execute_from_command_line
24 0.868 utility.execute()
24 0.868 File "/usr/local/lib/python3.8/dist-packages/django/core/management/init.py", line 395, in execute
24 0.868 django.setup()
24 0.868 File "/usr/local/lib/python3.8/dist-packages/django/init.py", line 24, in setup
24 0.869 apps.populate(settings.INSTALLED_APPS)
24 0.869 File "/usr/local/lib/python3.8/dist-packages/django/apps/registry.py", line 114, in populate
24 0.869 app_config.import_models()
24 0.869 File "/usr/local/lib/python3.8/dist-packages/django/apps/config.py", line 301, in import_models
24 0.869 self.models_module = import_module(models_module_name)
24 0.869 File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
24 0.869 return _bootstrap._gcd_import(name[level:], package, level)
24 0.869 File "", line 1014, in _gcd_import
24 0.869 File "", line 991, in _find_and_load
24 0.870 File "", line 975, in _find_and_load_unlocked
24 0.870 File "", line 671, in _load_unlocked
24 0.870 File "", line 783, in exec_module
24 0.870 File "", line 219, in _call_with_frames_removed
24 0.870 File "/webodm/app/models/init.py", line 1, in
24 0.870 from .image_upload import ImageUpload, image_directory_path
24 0.870 File "/webodm/app/models/image_upload.py", line 1, in
24 0.870 from .task import Task, assets_directory_path
24 0.870 File "/webodm/app/models/task.py", line 32, in
24 0.870 from app.cogeo import assure_cogeo
24 0.870 File "/webodm/app/cogeo.py", line 9, in
24 0.870 from rio_cogeo.cogeo import cog_validate, cog_translate
24 0.870 File "/usr/local/lib/python3.8/dist-packages/rio_cogeo/init.py", line 3, in
24 0.870 from rio_cogeo.cogeo import cog_info, cog_translate, cog_validate # noqa
24 0.870 File "/usr/local/lib/python3.8/dist-packages/rio_cogeo/cogeo.py", line 13, in
24 0.870 import morecantile
24 0.870 File "/usr/local/lib/python3.8/dist-packages/morecantile/init.py", line 14, in
24 0.871 from .defaults import tms # noqa
24 0.871 File "/usr/local/lib/python3.8/dist-packages/morecantile/defaults.py", line 11, in
24 0.871 from .models import TileMatrixSet
24 0.871 File "/usr/local/lib/python3.8/dist-packages/morecantile/models.py", line 25, in
24 0.871 WGS84_CRS = CRS.from_epsg(4326)
24 0.871 File "/usr/local/lib/python3.8/dist-packages/rasterio/crs.py", line 333, in from_epsg
24 0.871 obj._crs = _CRS.from_epsg(code)
24 0.871 File "rasterio/_crs.pyx", line 287, in rasterio._crs._CRS.from_epsg
24 0.873 rasterio.errors.CRSError: The EPSG code is unknown. PROJ: proj_create_from_database: SQLite error on SELECT name, type, coordinate_system_auth_name, coordinate_system_code, datum_auth_name, datum_code, area_of_use_auth_name, area_of_use_code, text_definition, deprecated FROM geodetic_crs WHERE auth_name = ? AND code = ?: no such column: area_of_use_auth_name

executor failed running [/bin/sh -c python manage.py collectstatic --noinput]: exit code: 1
ERROR: Service 'webapp' failed to build : Build failed

At this point I'm stuck, besides actually not knowing whether my attempts made any sense at all until here.

P.S.: I removed some # chars from the quotes as they got interpreted as headings.

[pierotofy]

WebODM doesn't have support for M1 as of yet. One would need to rebuild all docker images on M1 (and possibly add some M1 specific patches in the process). It hasn't been done yet.

[p-vitt]

That's what I tried to do. I would try to proceed, but I'm a little lost with this error from above:

rasterio.errors.CRSError: The EPSG code is unknown. PROJ: proj_create_from_database: SQLite error on SELECT name, type, coordinate_system_auth_name, coordinate_system_code, datum_auth_name, datum_code, area_of_use_auth_name, area_of_use_code, text_definition, deprecated FROM geodetic_crs WHERE auth_name = ? AND code = ?: no such column: area_of_use_auth_name

Do you have an idea what causes this problem, so that I gain insight into it and can proceed trying to get it running on an M1? As I said, I'm a little lost and don't know where to start looking for it.

[pierotofy]

I unfortunately don't have an M1, so I'm not sure why that error would appear :(

[p-vitt]

I'm sorry that I phrased that mistakeable. I'm not looking for the reason of this error but the part of ODM that could produce it so that I can narrow down the things I need to dig into to possibly find a solution.

[pierotofy]

Looks that it might be an error with the PROJ library; check that the PROJ database is correctly installed perhaps. app.cogeo.assure_cogeo is the function that is raising the error.

[Jeongyong-park]

This problem seems to be related to the cpu architecture of the Apple Silicon M1. The go runtime tries to execute some amd64 instruction, which is not present on the arm architecture.

As only the database seems to be affected, my first try was to update this container to a newer version:

diff --git a/db/Dockerfile b/db/Dockerfile
index 47d7278..fe766f5 100644
--- a/db/Dockerfile
+++ b/db/Dockerfile
@@ -1,7 +1,7 @@
-FROM postgres:9.5
+FROM postgres:13
MAINTAINER Piero Toffanin pt@masseranolabs.com
-ENV POSTGIS_MAJOR 2.3
+ENV POSTGIS_MAJOR 3
ENV POSTGRES_PASSWORD postgres
RUN . /etc/os-release \

This container seems to run, however, I wasn't able to get all the other containers up and running. A ./webodm.sh start just ran the old database image, not the new one, so I did a ./webodm.sh rebuild, which started to also rebuild the working images, just to fail on them. I'm quite new to docker, thus it can be due to my lack of knowledge. But as the webodm image was failing, I tried to update it's Dockerfile in the main folder as well:

diff --git a/Dockerfile b/Dockerfile
index fcd21b3..45c33db 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -4,6 +4,7 @@ MAINTAINER Piero Toffanin pt@masseranolabs.com
ENV PYTHONUNBUFFERED 1
ENV PYTHONPATH $PYTHONPATH:/webodm
ENV PROJ_LIB=/usr/share/proj
+ENV GDAL_VERSION=3.3.0
RUN mkdir /webodm
@@ -18,7 +19,7 @@ RUN wget --no-check-certificate https://deb.nodesource.com/setup_12.x -O /tmp/no
RUN apt-get -qq update && apt-get -qq install -y nodejs
-RUN apt-get -qq update && apt-get -qq install -y --no-install-recommends python3 python3-pip python3-setuptools python3-wheel git g++ python3-dev python2.7-dev libpq-dev binutils libproj-dev gdal-bin python3-gdal nginx certbot grass-core gettext-base cron postgresql-client-12 gettext
+RUN apt-get -qq update && apt-get -qq install -y --no-install-recommends python3 python3-pip python3-setuptools python3-wheel git g++ python3-dev python2.7-dev libpq-dev binutils libproj-dev gdal-bin libgdal-dev python3-gdal nginx certbot grass-core gettext-base cron postgresql-client-12 gettext
RUN update-alternatives --install /usr/bin/python python /usr/bin/python2.7 1 && update-alternatives --install /usr/bin/python python /usr/bin/python3.8 2

But this leaves me with an error in building the container:

24 0.867

24 0.867 During handling of the above exception, another exception occurred:
24 0.867
24 0.867 Traceback (most recent call last):
24 0.867 File "manage.py", line 22, in
24 0.867 execute_from_command_line(sys.argv)
24 0.867 File "/usr/local/lib/python3.8/dist-packages/django/core/management/init.py", line 419, in >execute_from_command_line
24 0.868 utility.execute()
24 0.868 File "/usr/local/lib/python3.8/dist-packages/django/core/management/init.py", line 395, in execute
24 0.868 django.setup()
24 0.868 File "/usr/local/lib/python3.8/dist-packages/django/init.py", line 24, in setup
24 0.869 apps.populate(settings.INSTALLED_APPS)
24 0.869 File "/usr/local/lib/python3.8/dist-packages/django/apps/registry.py", line 114, in populate
24 0.869 app_config.import_models()
24 0.869 File "/usr/local/lib/python3.8/dist-packages/django/apps/config.py", line 301, in import_models
24 0.869 self.models_module = import_module(models_module_name)
24 0.869 File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
24 0.869 return _bootstrap._gcd_import(name[level:], package, level)
24 0.869 File "", line 1014, in _gcd_import
24 0.869 File "", line 991, in _find_and_load
24 0.870 File "", line 975, in _find_and_load_unlocked
24 0.870 File "", line 671, in _load_unlocked
24 0.870 File "", line 783, in exec_module
24 0.870 File "", line 219, in _call_with_frames_removed
24 0.870 File "/webodm/app/models/init.py", line 1, in
24 0.870 from .image_upload import ImageUpload, image_directory_path
24 0.870 File "/webodm/app/models/image_upload.py", line 1, in
24 0.870 from .task import Task, assets_directory_path
24 0.870 File "/webodm/app/models/task.py", line 32, in
24 0.870 from app.cogeo import assure_cogeo
24 0.870 File "/webodm/app/cogeo.py", line 9, in
24 0.870 from rio_cogeo.cogeo import cog_validate, cog_translate
24 0.870 File "/usr/local/lib/python3.8/dist-packages/rio_cogeo/init.py", line 3, in
24 0.870 from rio_cogeo.cogeo import cog_info, cog_translate, cog_validate # noqa
24 0.870 File "/usr/local/lib/python3.8/dist-packages/rio_cogeo/cogeo.py", line 13, in
24 0.870 import morecantile
24 0.870 File "/usr/local/lib/python3.8/dist-packages/morecantile/init.py", line 14, in
24 0.871 from .defaults import tms # noqa
24 0.871 File "/usr/local/lib/python3.8/dist-packages/morecantile/defaults.py", line 11, in
24 0.871 from .models import TileMatrixSet
24 0.871 File "/usr/local/lib/python3.8/dist-packages/morecantile/models.py", line 25, in
24 0.871 WGS84_CRS = CRS.from_epsg(4326)
24 0.871 File "/usr/local/lib/python3.8/dist-packages/rasterio/crs.py", line 333, in from_epsg
24 0.871 obj._crs = _CRS.from_epsg(code)
24 0.871 File "rasterio/_crs.pyx", line 287, in rasterio._crs._CRS.from_epsg
24 0.873 rasterio.errors.CRSError: The EPSG code is unknown. PROJ: proj_create_from_database: SQLite error on SELECT name, type, coordinate_system_auth_name, coordinate_system_code, datum_auth_name, datum_code, area_of_use_auth_name, area_of_use_code, text_definition, deprecated FROM geodetic_crs WHERE auth_name = ? AND code = ?: no such column: area_of_use_auth_name
executor failed running [/bin/sh -c python manage.py collectstatic --noinput]: exit code: 1
ERROR: Service 'webapp' failed to build : Build failed

At this point I'm stuck, besides actually not knowing whether my attempts made any sense at all until here.

P.S.: I removed some # chars from the quotes as they got interpreted as headings.

node js also support M1 after version 15.
that's why I think change https://deb.nodesource.com/setup_12.x to https://deb.nodesource.com/setup_16.x
and python also arm64 support after 3.9.1

[p-vitt]

node js also support M1 after version 15.
that's why I think change https://deb.nodesource.com/setup_12.x to https://deb.nodesource.com/setup_16.x
and python also arm64 support after 3.9.1

Thank you for the hints. I already switched to Py3.9, but the Node.js change I have to keep in mind.

Actually, I changed the plan a little bit. Right now I'm trying to get ODM running natively on the M1. Once this is done, I want to adopt those changes to/learn from those changes for WebODM. The problem I'm facing right now is to build OpenSfm, which is still failing in the configure step. I think this is due to the glibc, but this is just a rough guess. Still digging...

[Jeongyong-park]

Wow, I'm looking forward to it. 👍🏻 2021년 6월 17일 (목) 오후 7:40, pvitt ***@***.***>님이 작성:

…

node js also support M1 after version 15. that's why I think change https://deb.nodesource.com/setup_12.x to https://deb.nodesource.com/setup_16.x and python also arm64 support after 3.9.1 Thank you for the hints. I already switched to Py3.9, but the Node.js change I have to keep in mind. Actually, I changed the plan a little bit. Right now I'm trying to get ODM running natively on the M1. Once this is done, I want to adopt those changes to/learn from those changes for WebODM. The problem I'm facing right now is to build OpenSfm, which is still failing in the configure step. I think this is due to the glibc, but this is just a rough guess. Still digging... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#997 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPGQ6ETCMIGKSCJ5GLPOWDTTHGKVANCNFSM46OIYT2Q> .

[pierotofy]

A fund is now open toward this effort. https://fund.webodm.org/#/fund/6

[shawndearmond]

Looks like the effort is funded. Hooray!

Any update on getting this fixed?

[pierotofy]

The effort was completed, so WebODM should work on M1 now (forgot to close this issue)! Just make sure to update via ./webodm.sh update && ./webodm.sh restart before running. 👏

[shawndearmond]

It works! Thank you!!