Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Request to add pdlib to PECL. #56

Open
apoorv569 opened this issue Dec 19, 2022 · 72 comments · Fixed by nextcloud/all-in-one#3999
Open

Request to add pdlib to PECL. #56

apoorv569 opened this issue Dec 19, 2022 · 72 comments · Fixed by nextcloud/all-in-one#3999

Comments

@apoorv569
Copy link

I am using Nextcloud-AIO and want to install the Face Recognition app which requires this extension as a dependency. Please add this to PECL so me and anybody else can use this.

I got recommended to request this here,
nextcloud/all-in-one#1551
@sebastiencastermans
Copy link

Hi, I have the same issue and... the same request! I vote for that.

@zybster
Copy link

zybster commented Dec 23, 2022

Would really be great if this could be added - FaceRecognition is a very nice addon

@antmarra
Copy link

antmarra commented Jan 8, 2023

Please add pdlib to AIO, or let us know how to do this.

@froman753
Copy link

Same request, definitely vote for this!

@enoch85
Copy link

enoch85 commented Jan 20, 2023

It's also used in the Nextcloud VM but that repo doesn't work. Would be super useful with "an official" release from something like PECL.

@rastographics
Copy link

Same issue...I want to install face recognition on nextcloud AIO so adding my vote to this.

@tola5
Copy link

tola5 commented Jan 21, 2023

one more plus from here

@enoch85
Copy link

enoch85 commented Jan 21, 2023 •
edited
Loading

Please stop adding your "+1" comments, it notifies all that's posted here, or that subscribes to this issue. Add your 👍 in the first post instead.

Thanks!

@stalsbyt
Copy link

+1

@enoch85
Copy link

enoch85 commented Jan 25, 2023

Please stop adding your "+1" comments, it notifies all that's posted here, or that subscribes to this issue. Add your +1 in the first post instead.

Thanks!

@stalsbyt PLEASE! :)

@JeCheeseSmith
Copy link

I am using Nextcloud-AIO and want to install the Face Recognition app which requires this extension as a dependency. Please add this to PECL so me and anybody else can use this.

I got recommended to request this here, nextcloud/all-in-one#1551

I would like to use Face Recognition too on Nextcloud AIO. I'm a newbie to any of this but I would like too use the API of Face Recognition to build a genetic tree with the faces found.

@szaimen
Copy link

szaimen commented Feb 10, 2023

cc @stalker314314 @matiasdelellis what is your take on this?

@artur90226
Copy link

+1

@jrmarincallejero
Copy link

Un voto mas

@matiasdelellis
Copy link
Contributor

Doubt.. I'm no docker expert... 🤔

Why does it seem so complicated to do something like this?

RUN wget https://github.com/goodspb/pdlib/archive/master.zip \
  && mkdir -p /usr/src/php/ext/ \
  && unzip -d /usr/src/php/ext/ master.zip \
  && rm master.zip
RUN docker-php-ext-install pdlib-master

You could also install dlib compiling, and it would be important to do it, (to optimize it for hardware), with 5 more lines of code.

ps: https://github.com/matiasdelellis/facerecognition/wiki/Docker

@Zoey2936
Copy link

If we included all dependencies ever requested in AIO, then Nextcloud AIO would be unnecessary big after some time and installing decencies by default would mean additionally Things to maintain and the docker built could easily break. That is the reason AIO only supports adding PHP extension using pecl. And also is nearly every PHP extension requested until now (except this extension here) available over pecl. (nextcloud/all-in-one#1159 (comment))

@matiasdelellis
Copy link
Contributor

Hello everyone,
Personally, I see Docker as a way to do dirty stuff in a safe environment, and keep my server clean. So the argument doesn't convince me... 🤦🏻 🙈

I don't know.. I'm probably wrong.. 🤔

However, I think there is an opportunity for improvement in the optimization of the extension, and that is worth it. 😄 😬

Ok okey.. Well.
During the day, I was doing tests, and I was able to compile dlib as a submodule of the same extension.

So, both dlib and the extension compile as native architecture, and are optimized for the hardware used.

About PECL, which is the main issue here, I'm not sure if I can do it, since I don't own the extension, and would prefer not to keep a fork, but in the meantime I invite you to try the changes since it will be essential to be build as a PECL package.

@matiasdelellis
Copy link
Contributor

Well,
I guess I have something interesting... As I said, I will not upload to pecl soon.. but I have a working package to test and probably just use.. 😄

sudo apk install libpng openblas lapack
wget https://github.com/matiasdelellis/pdlib/releases/download/v1.1.0/pdlib-1.1.0.tgz
sudo pecl install ./pdlib-1.1.0.tgz

In short, you must install dependencies (NEXTCLOUD_ADDITIONAL_APKS 😉 😉 wink...), download the package 🤷🏻‍♂️ , and install it from package..

It's not the same as before???!?!??? 🙈 😅 😅 😆

Well, now being dlib in the same code, many problems are minimized, and it is always optimized when compiling it.. 😉

So, you can adapt the AIO scripts to download the tarball and install them???

I imagine something like that.. .
-e NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS="https://github.com/matiasdelellis/pdlib/releases/download/v1.1.0/pdlib-1.1.0.tgz"

similar to what shivammathur/setup-php@v2 does.. 🤔

@Zoey2936
Copy link

Zoey2936 commented Mar 2, 2023

@szaimen could you please have a look?

@matiasdelellis
Copy link
Contributor

matiasdelellis commented Mar 2, 2023 •
edited
Loading

Hi everyone,

Maybe not even have to change the scripts, but I need someone to test it.
Pecl can install the package from a URL.. 🎉

[matias@nube ~]$ sudo pecl install https://services.delellis.com.ar/data/facerecognition/pdlib-1.1.0.tgz
downloading pdlib-1.1.0.tgz ...
Starting to download pdlib-1.1.0.tgz (3,078,356 bytes)
.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done: 3,078,356 bytes
WARNING: channel "pecl.php.net" has updated its protocols, use "pecl channel-update pecl.php.net" to update
1389 source files, building
running: phpize
Configuring for:
PHP Api Version:         20200930
Zend Module Api No:      20200930
Zend Extension Api No:   420200930
building in /var/tmp/pear-build-rootGQ6ovZ/pdlib-1.1.0
running: /var/tmp/pdlib/configure --with-php-config=/usr/bin/php-config
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E

....

So, -e NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS="https://services.delellis.com.ar/data/facerecognition/pdlib-1.1.0.tgz", probably works directly. 😄

EDIT: Change the url to a private server, since PECL doesn't like github redirects. 😅

EDIT 2: It will fail, because it does some character-only validation.
https://github.com/nextcloud/all-in-one/blob/c23b7aa3969a5f029e3cc9f9abc6be34e97a9ec0/Containers/mastercontainer/start.sh#L207-L212

Beyond that, I would think it should work. 🤔

@szaimen
Copy link

szaimen commented Mar 3, 2023

Wow, thanks a lot for this nice new aproach @matiasdelellis ! :)

I wonder how much work would it be to publish this now on PECL? With that it would work out-of-the-box in AIO directly and would be prefered from a UX perspective since the user would not need to maintain and change this link when a new release comes out.

As for libpng openblas lapack are these runtime dependencies or only needed for compiling?

@goodspb
Copy link
Owner

goodspb commented Mar 3, 2023

sorry guys, I lost track of PHP now. Plz give me some time to figure out how to upload it to PECL. 😭

@matiasdelellis
Copy link
Contributor

Hi @goodspb
If you're willing to upload to PECL (Theoretically, it shouldn't be difficult to be accepted.. and you have a lot of support from here.. 😬), I can do a pull request with the changes.

I would love to help you to improve and extend it, but I don't want to be responsible for another project forking this. 😄

About the current proposal:

  • The main change is that dlib is added as a git submodule to the repository. This allows to minimize many common problems of this library. (Old versions, compilations with X11 or without that, library without headers, etc etc, etc.). In general distribution problems, but that are only problematic for us.
  • Again... this greatly simplifies the compilation of the project, and greatly minimizes some necessary explanations or discussions. 😅 And for me the most important thing, it will always be optimized. 😅
  • Add the tests using github actions. Spoiler 🙊 , I don't have any server with php 8.2, but we can say that it passes the tests and it should work.. 😬

My TODO list about that:

  • We could do it without relying on libpng, openblas and lapack since dlib includes some copies as well to avoid problems. But I'd like to make it optional. I have to think about it and test before. Meanwhile, these dependencies are easier to find, and are well compiled by the distributions.
  • Also, I'd still like to be able to compile against the distribution's dlib, since the compilation would be much faster. Less optimized, but faster compilation at last. 😅
  • I have to review the script that generates the package.xml to the add dlib licenses, and some other files. Theoretically this is easy.. 🤔
  • With this approach we can't get the dlib version easily. So I commented it in the module info, however, it's less relevant now. 🤷🏻‍♂️

Now I don't remember anything else relevant to this discussion.. 🤔

Well,
If you want, I start the PR.. 😃

This was referenced Mar 11, 2023
Open
Closed
@Zoey2936
Copy link

Zoey2936 commented Mar 18, 2023 •
edited
Loading

@matiasdelellis I've tried installing it from inside the nextcloud-aio-nextcloud container, but it does not work (I've installed build-base and autoconf already):

log 
bash-5.1# pecl install https://services.delellis.com.ar/data/facerecognition/pdlib-1.1.0.tgz
downloading pdlib-1.1.0.tgz ...
Starting to download pdlib-1.1.0.tgz (3,078,356 bytes)
.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done: 3,078,356 bytes
1389 source files, building
running: phpize
Configuring for:
PHP Api Version:         20200930
Zend Module Api No:      20200930
Zend Extension Api No:   420200930
configure.ac:18: warning: $as_echo is obsolete; use AS_ECHO(["message"]) instead
build/php.m4:2072: PHP_CONFIG_NICE is expanded from...
configure.ac:18: the top level
configure.ac:161: warning: The macro `AC_LANG_C' is obsolete.
configure.ac:161: You should run autoupdate.
./lib/autoconf/c.m4:72: AC_LANG_C is expanded from...
build/libtool.m4:2728: _LT_AC_LANG_C_CONFIG is expanded from...
build/libtool.m4:2727: AC_LIBTOOL_LANG_C_CONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: The macro `AC_LANG_C' is obsolete.
configure.ac:161: You should run autoupdate.
./lib/autoconf/c.m4:72: AC_LANG_C is expanded from...
lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
./lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded from...
build/libtool.m4:561: _LT_AC_LOCK is expanded from...
build/libtool.m4:1185: AC_LIBTOOL_SYS_HARD_LINK_LOCKS is expanded from...
build/libtool.m4:2728: _LT_AC_LANG_C_CONFIG is expanded from...
build/libtool.m4:2727: AC_LIBTOOL_LANG_C_CONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: The macro `AC_TRY_LINK' is obsolete.
configure.ac:161: You should run autoupdate.
./lib/autoconf/general.m4:2920: AC_TRY_LINK is expanded from...
lib/m4sugar/m4sh.m4:692: _AS_IF_ELSE is expanded from...
lib/m4sugar/m4sh.m4:699: AS_IF is expanded from...
./lib/autoconf/general.m4:2249: AC_CACHE_VAL is expanded from...
./lib/autoconf/general.m4:2270: AC_CACHE_CHECK is expanded from...
build/libtool.m4:561: _LT_AC_LOCK is expanded from...
build/libtool.m4:1185: AC_LIBTOOL_SYS_HARD_LINK_LOCKS is expanded from...
build/libtool.m4:2728: _LT_AC_LANG_C_CONFIG is expanded from...
build/libtool.m4:2727: AC_LIBTOOL_LANG_C_CONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me:${as_lineno-$LINENO}: WARNING: \`$CC' does not support \`-c -o', so \`make -j' may be unsafe
build/libtool.m4:1185: AC_LIBTOOL_SYS_HARD_LINK_LOCKS is expanded from...
build/libtool.m4:2728: _LT_AC_LANG_C_CONFIG is expanded from...
build/libtool.m4:2727: AC_LIBTOOL_LANG_C_CONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me: WARNING: \`$CC' does not support \`-c -o', so \`make -j' may be unsafe
build/libtool.m4:1185: AC_LIBTOOL_SYS_HARD_LINK_LOCKS is expanded from...
build/libtool.m4:2728: _LT_AC_LANG_C_CONFIG is expanded from...
build/libtool.m4:2727: AC_LIBTOOL_LANG_C_CONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me:${as_lineno-$LINENO}: WARNING: output file \`$ofile' does not exist
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me: WARNING: output file \`$ofile' does not exist
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me:${as_lineno-$LINENO}: WARNING: output file \`$ofile' does not look like a libtool script
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me: WARNING: output file \`$ofile' does not look like a libtool script
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me:${as_lineno-$LINENO}: WARNING: using \`LTCC=$LTCC', extracted from \`$ofile'
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me: WARNING: using \`LTCC=$LTCC', extracted from \`$ofile'
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: tag name \"$tagname\" already exists
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: The macro `AC_LANG_CPLUSPLUS' is obsolete.
configure.ac:161: You should run autoupdate.
./lib/autoconf/c.m4:262: AC_LANG_CPLUSPLUS is expanded from...
build/libtool.m4:2810: _LT_AC_LANG_CXX_CONFIG is expanded from...
build/libtool.m4:2809: AC_LIBTOOL_LANG_CXX_CONFIG is expanded from...
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me:${as_lineno-$LINENO}: WARNING: \`$CC' does not support \`-c -o', so \`make -j' may be unsafe
build/libtool.m4:1185: AC_LIBTOOL_SYS_HARD_LINK_LOCKS is expanded from...
build/libtool.m4:2810: _LT_AC_LANG_CXX_CONFIG is expanded from...
build/libtool.m4:2809: AC_LIBTOOL_LANG_CXX_CONFIG is expanded from...
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
configure.ac:161: warning: back quotes and double quotes must not be escaped in: $as_me: WARNING: \`$CC' does not support \`-c -o', so \`make -j' may be unsafe
build/libtool.m4:1185: AC_LIBTOOL_SYS_HARD_LINK_LOCKS is expanded from...
build/libtool.m4:2810: _LT_AC_LANG_CXX_CONFIG is expanded from...
build/libtool.m4:2809: AC_LIBTOOL_LANG_CXX_CONFIG is expanded from...
build/libtool.m4:1918: _LT_AC_TAGCONFIG is expanded from...
build/libtool.m4:70: AC_LIBTOOL_SETUP is expanded from...
build/libtool.m4:52: _AC_PROG_LIBTOOL is expanded from...
build/libtool.m4:39: AC_PROG_LIBTOOL is expanded from...
configure.ac:161: the top level
building in /tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0
running: /tmp/pear/temp/pdlib/configure --with-php-config=/usr/local/bin/php-config
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for a sed that does not truncate output... /bin/sed
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for cc... cc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether cc accepts -g... yes
checking for cc option to enable C11 features... none needed
checking how to run the C preprocessor... cc -E
checking for icc... no
checking for suncc... no
checking for system library directory... lib
checking if compiler supports -Wl,-rpath,... yes
checking build system type... x86_64-pc-linux-musl
checking host system type... x86_64-pc-linux-musl
checking target system type... x86_64-pc-linux-musl
checking for PHP prefix... /usr/local
checking for PHP includes... -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib
checking for PHP extension directory... /usr/local/lib/php/extensions/no-debug-non-zts-20200930
checking for PHP installed headers prefix... /usr/local/include/php
checking if debug is enabled... no
checking if zts is enabled... no
checking for gawk... no
checking for nawk... no
checking for awk... awk
checking if awk is broken... no
checking for pdlib support... yes, shared
checking for g++... g++
checking whether the compiler supports GNU C++... yes
checking whether g++ accepts -g... yes
checking for g++ option to enable C++11 features... none needed
checking how to run the C++ preprocessor... g++ -E
checking for a sed that does not truncate output... /bin/sed
checking for ld used by cc... /usr/x86_64-alpine-linux-musl/bin/ld
checking if the linker (/usr/x86_64-alpine-linux-musl/bin/ld) is GNU ld... yes
checking for /usr/x86_64-alpine-linux-musl/bin/ld option to reload object files... -r
checking for BSD-compatible nm... /usr/bin/nm -B
checking whether ln -s works... yes
checking how to recognize dependent libraries... pass_all
checking for stdio.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for strings.h... yes
checking for sys/stat.h... yes
checking for sys/types.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking how to run the C++ preprocessor... g++ -E
checking the maximum length of command line arguments... 98304
checking command to parse /usr/bin/nm -B output from cc object... ok
checking for objdir... .libs
checking for ar... ar
checking for ranlib... ranlib
checking for strip... strip
checking if cc supports -fno-rtti -fno-exceptions... no
checking for cc option to produce PIC... -fPIC
checking if cc PIC flag -fPIC works... yes
checking if cc static flag -static works... yes
checking if cc supports -c -o file.o... yes
checking whether the cc linker (/usr/x86_64-alpine-linux-musl/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... no

creating libtool
appending configuration tag "CXX" to libtool
checking for ld used by g++... /usr/x86_64-alpine-linux-musl/bin/ld -m elf_x86_64
checking if the linker (/usr/x86_64-alpine-linux-musl/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the g++ linker (/usr/x86_64-alpine-linux-musl/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC
checking if g++ PIC flag -fPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking whether the g++ linker (/usr/x86_64-alpine-linux-musl/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
(cached) (cached) checking how to hardcode library paths into programs... immediate
configure: patching config.h.in
configure: creating ./config.status

Build configuration for PDlib v1.1.0 done correctly.

  dlib version:

  CXXFLAGS    : -g -O3 -march=native
  LDFLAGS     :
  LIBDIR:     :
  LIBADD:     :  -lstdc++ -lopenblas -llapack -lpng

Please submit bug reports at:
  https://github.com/goodspb/pdlib/issues

config.status: creating config.h
running: make
/bin/sh /tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/libtool --mode=compile g++ -I. -I/tmp/pear/temp/pdlib -I/tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/include -I/tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/main -I/tmp/pear/temp/pdlib -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -I/tmp/pear/temp/pdlib/dlib  -DHAVE_CONFIG_H  -g -O3 -march=native   -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -DDLIB_NO_GUI_SUPPORT=1 -DDLIB_USE_BLAS -DDLIB_USE_LAPACK -DDLIB_PNG_SUPPORT -c /tmp/pear/temp/pdlib/pdlib.cc -o pdlib.lo
mkdir .libs
 g++ -I. -I/tmp/pear/temp/pdlib -I/tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/include -I/tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/main -I/tmp/pear/temp/pdlib -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -I/tmp/pear/temp/pdlib/dlib -DHAVE_CONFIG_H -g -O3 -march=native -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -DDLIB_NO_GUI_SUPPORT=1 -DDLIB_USE_BLAS -DDLIB_USE_LAPACK -DDLIB_PNG_SUPPORT -c /tmp/pear/temp/pdlib/pdlib.cc  -fPIC -DPIC -o .libs/pdlib.o
In file included from /usr/local/include/php/Zend/zend_portability.h:58,
                 from /usr/local/include/php/Zend/zend_types.h:25,
                 from /usr/local/include/php/Zend/zend.h:27,
                 from /usr/local/include/php/main/php.h:31,
                 from /tmp/pear/temp/pdlib/pdlib.cc:26:
/tmp/pear/temp/pdlib/pdlib.cc: In function 'void php_face_recognition_free(zend_object*)':
/tmp/pear/temp/pdlib/pdlib.cc:139:79: warning: 'offsetof' within non-standard-layout type 'face_recognition' {aka '_face_recognition'} is conditionally-supported [-Winvalid-offsetof]
  139 |         face_recognition *fr = (face_recognition*)((char*)object - XtOffsetOf(face_recognition, std));
/tmp/pear/temp/pdlib/pdlib.cc:139:68: note: in expansion of macro 'XtOffsetOf'
  139 |         face_recognition *fr = (face_recognition*)((char*)object - XtOffsetOf(face_recognition, std));
      |                                                                    ^~~~~~~~~~
/tmp/pear/temp/pdlib/pdlib.cc: In function 'zend_result zm_startup_pdlib(int, int)':
/tmp/pear/temp/pdlib/pdlib.cc:173:59: warning: 'offsetof' within non-standard-layout type 'face_recognition' {aka '_face_recognition'} is conditionally-supported [-Winvalid-offsetof]
  173 |         face_recognition_obj_handlers.offset = XtOffsetOf(face_recognition, std);
/tmp/pear/temp/pdlib/pdlib.cc:173:48: note: in expansion of macro 'XtOffsetOf'
  173 |         face_recognition_obj_handlers.offset = XtOffsetOf(face_recognition, std);
      |                                                ^~~~~~~~~~
/bin/sh /tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/libtool --mode=compile g++ -I. -I/tmp/pear/temp/pdlib -I/tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/include -I/tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/main -I/tmp/pear/temp/pdlib -I/usr/local/include/php -I/usr/local/include/php/main -I/usr/local/include/php/TSRM -I/usr/local/include/php/Zend -I/usr/local/include/php/ext -I/usr/local/include/php/ext/date/lib -I/tmp/pear/temp/pdlib/dlib  -DHAVE_CONFIG_H  -g -O3 -march=native   -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -DDLIB_NO_GUI_SUPPORT=1 -DDLIB_USE_BLAS -DDLIB_USE_LAPACK -DDLIB_PNG_SUPPORT -c /tmp/pear/temp/pdlib/dlib/dlib/all/source.cpp -o dlib/dlib/all/source.lo
/tmp/pear/temp/pear-build-defaultuserFHlCda/pdlib-1.1.0/libtool: line 1290: can't create dlib/dlib/all/source.loT: nonexistent directory
mkdir dlib/dlib/all/.libs
mkdir: cannot create directory 'dlib/dlib/all/.libs': No such file or directory
make: *** [Makefile:211: dlib/dlib/all/source.lo] Error 1
ERROR: `make' failed

@samssausages
Copy link

+1, want it for Nexcloud AIO support

@likt0r
Copy link

likt0r commented Mar 27, 2023

+1 fro support in AIO would be great, thx for your work :)

@wuast94
Copy link

wuast94 commented Apr 14, 2023

+1 for support

1 similar comment
@johnnyq
Copy link

johnnyq commented Apr 17, 2023

+1 for support

@szaimen
Copy link

szaimen commented Aug 24, 2023 •
edited
Loading

The php memory limit in AIO can be adjusted like so: https://github.com/nextcloud/all-in-one#how-to-adjust-the-php-memory-limit-for-nextcloud

Also see https://github.com/nextcloud/all-in-one#how-to-run-occ-commands how to run occ commands in AIO.

@sunnyd24
Copy link

sunnyd24 commented Aug 25, 2023 •
edited
Loading

The php memory limit in AIO can be adjusted like so: https://github.com/nextcloud/all-in-one#how-to-adjust-the-php-memory-limit-for-nextcloud

Also see https://github.com/nextcloud/all-in-one#how-to-run-occ-commands how to run occ commands in AIO.

@szaimen Thanks!

For others with similar AIO related questions OR new to face recognition here's a quick start guide for us AIO users which took me a while to figure out:

NEXTCLOUD MASTER CONTAINER CONFIG
=============================

Open Nextcloud AIO interface, and stop containers.

docker stop nextcloud-aio-mastercontainer
docker rm nextcloud-aio-mastercontainer

Amend the PHP memory limit to be anything from 1024M to 2048M should suffice in --env NEXTCLOUD_MEMORY_LIMIT=4096M \ to docker run command. I used 4096M as I was unsure how much is required.

docker run [your amended multi-line docker run command with updated PHP memory limit]

Open Nextcloud AIO interface, and start containers.

FASTEST METHOD using Docker
======================
docker build -t facerecognition https://github.com/matiasdelellis/facerecognition-external-model.git

openssl rand -base64 32
echo EnTeRYoURgEnErAtEdLoNgKeYhErE= > api.key
NOTE: change the key above generated in output of openssl rand -base64 32 and copy it to clipboard.

sudo docker run --rm --detach -i -p 8080:5000 -e API_KEY="EnTeRYoURgEnErAtEdLoNgKeYhErE" facerecognition
NOTE: change 8080 to any external port available on host PC where Docker is running

CONFIG
======
sudo docker run -it --rm --volume nextcloud_aio_nextcloud:/var/www/html:rw alpine sh -c "apk add --no-cache nano && nano /var/www/html/config/config.php"

$CONFIG = array (
  ...........................................
  'facerecognition.external_model_url' => 'http://192.168.1.52:8080',
  'facerecognition.external_model_api_key' => 'EnTeRYoURgEnErAtEdLoNgKeYhErE',
  ...............................
);

NOTE: Add the above lines starting with 'facerecognition' within array in config.php, change host IP address and port, and be careful edigint his file as this can break Nextcloud from starting if done incorrectly.

sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ face:setup -m 5
sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ face:setup -M 4G

NOTE: change 4G to be anything between 2G and 4G of memory.

sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ face:setup
NOTE: just checks configuration and provides a table to see that settings have been applied.

Go here for advanced settings such as enabling face recognition on group/external/shared folders

USAGE
=====
sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ face:background_job --user_id USERNAME
NOTE: change USERNAME to your nextcloud username you want to run face recognition on. You can also append --timeout 18000, to stop after 5 hours. However, for first run you may want to omit timeout as for my initial library test of 573 photos, it took approx 7 hours or so to complete, that's approx 44 seconds per photo (mind you I did have my temporary files settings at max of 3325x2494 (4x3))!

sudo docker exec --user www-data -it nextcloud-aio-nextcloud php occ user:setting USERNAME facerecognition full_image_scan_done false
NOTE: Another useful command to have if you need to start a full image scan from scratch due to advanced settings you have changed, it will not do it automatically! Remember to change USERNAME to your nextcloud username you want to to rescan.

REFERENCE
=========

For reference, I followed this generic guide with some amendments listed in steps above making it ready for an AIO install:

**Apologies if I've made any errors in the above, it's the best I could capture on the go, feel free to let me know and I will update the above for other users **
Thanks @szaimen for the PHP memory limit commands reminder!

@szaimen
Copy link

szaimen commented Aug 25, 2023

Hi, thanks for this guide! However I think you missed actually adjusting the php memory limit for the Nextcloud container itself or is this not need to be adjusted when using the external model @matiasdelellis?

If it is, see https://github.com/nextcloud/all-in-one#how-to-adjust-the-php-memory-limit-for-nextcloud

@matiasdelellis
Copy link
Contributor

In principle, the external model only requires 128 MB of ram, since the large memory consumption is done by the analysis of the images (specifically the search for faces) and another process does it.. However, the clustering of faces may require more memory.. I don't have much statistics about it, but there are reports of 2 GB with 50 thousand faces of single user.

@szaimen
Copy link

szaimen commented Aug 26, 2023 •
edited
Loading

Btw another advice: @sunnyd24 you could also add the container to the nextcloud-aio network. Then you do not need to expose the sevice on the host and instead can point directly at the container name...

@szaimen
Copy link

szaimen commented Aug 26, 2023 •
edited
Loading

Also my personal recommendation for the analysis_image_area: https://github.com/nextcloud/vm/blob/5a245496ce055d5c1697b689ede6e398548f7603/old/face-recognition.sh#L138

@Csantos88
Copy link

+1 Vote for add pdlib in PECL PHP Repos

@goodspb
Copy link
Owner

goodspb commented Sep 3, 2023

@matiasdelellis really grand that you could help with this. thanks.

I have written to PECL but never got feedback. I don't know why and don't know how to proceed. >.<

@boboche
Copy link

boboche commented Sep 27, 2023

+1, its 2023, there's no open source DAM with SIMPLE install, automatic AI object tagging / classification that simply installs and work out of the box without having to fight with some docker-fu or configuration headaches. Nextcloud is a super nice package and has everything in place to go next level.

But what should be a simple add-on from the app store + some simple post-config (i.e. select which model to download/apply + add. cfg, so that it doesn't make a huge VM or initial download for people who don't need it in the first place) is a in-shell php/snap/scripting snafu. Great and "normal" for coders, unfortunately not for the intended target audience tho.

@szaimen
Copy link

szaimen commented Sep 28, 2023

Hi, I just wanted to mention that AiO has this now: https://github.com/nextcloud/all-in-one/tree/main/community-containers#how-to-add-containers. So the community could potentially add the facerecognition container as additional container there. Feel free to ping me if you should need help on this!

@PrivatHomeServer
Copy link

Hello Simon,
I have watched your activities regarding the community containers on Friday. Looks very promising. I'm not the absolute IT expert - so I went the way of @sunnyd24 and finished part 1 with the memory allocation. Now encouraged by your post @szaimen I wanted to look at what we need to do to get community docker for "facerecognition". For the other json files, I looked once. "image ".../..." --> is missing in this case since we always run a build (docker build -t facerecognition https://github.com/matiasdelellis/facerecognition-external-model.git)

What do we need to do to make it a community docker here now? For me still the only issue with Nextcloud AOI, because the "regognize" solution is not usable for faces.

I would like to see a way here.

@szaimen
Copy link

szaimen commented Oct 1, 2023 •
edited
Loading

"image ".../..." --> is missing in this case since we always run a build (docker build -t facerecognition https://github.com/matiasdelellis/facerecognition-external-model.git)

What do we need to do to make it a community docker here now?

Indeed someone would need to create an image out of it and publish it on docker hub.

For that you might have a look at https://github.com/szaimen/aio-fail2ban/blob/main/.github/workflows/docker-build.yml and submit such a workflow at https://github.com/matiasdelellis/facerecognition-external-model.git

Regarding the json, I just leave this here as an example:

{
    "aio_services_v1": [
        {
            "container_name": "nextcloud-aio-facerecognition",
            "display_name": "Computing container for facerecognition",
            "documentation": "https://github.com/nextcloud/all-in-one/tree/main/community-containers/facerecognition",
            "image": "matiasdelellis/aio-facerecognition",
            "image_tag": "v1",
            "internal_port": "5000",
            "restart": "unless-stopped",
            "environment": [
                "TZ=%TIMEZONE%",
                "API_KEY=some-super-secret-api-key"
            ],
            "aio_variables": [
                "nextcloud_memory_limit=4096M"
            ],
            "nextcloud_exec_commands": [
                "php /var/www/html/occ app:install facerecognition",
                "php /var/www/html/occ config:system:set facerecognition.external_model_url --value nextcloud-aio-facerecognition:5000",
                "php /var/www/html/occ config:system:set facerecognition.external_model_api_key --value some-super-secret-api-key",
                "php /var/www/html/occ face:setup -m 5",
                "php /var/www/html/occ face:setup -M 4G",
                "php /var/www/html/occ occ face:background_job &"
            ]
        }
    ]
}

@PrivatHomeServer
Copy link

@szaimen thank you very much for your comments. Superficially I have understood the topic. However, I am too little IT-person to be able to really implement it. Unfortunately.

From my point of view, this add-on is essential for the photo application to stand out against solutions like immich.

@szaimen szaimen mentioned this issue Oct 19, 2023
Merged
@benji0522
Copy link

wanted

@marcselis
Copy link

+1

@Sven286
Copy link

Sven286 commented Dec 9, 2023

Because it all took more than a year to even have a glimph of face recognition feature. In the mean time a lot of development happened at Immich. I switched back to there. Good luck with nextcloud.
/unsubscribe

@elvismercado
Copy link

+1

2 similar comments
@gabglus
Copy link

gabglus commented Dec 28, 2023

+1

@otola
Copy link

otola commented Dec 30, 2023

+1

@nbluemer
Copy link

I also need pdlib in order to run face recognition on Nextcloud AIO.

@rastographics
Copy link

Because it all took more than a year to even have a glimph of face recognition feature. In the mean time a lot of development happened at Immich. I switched back to there. Good luck with nextcloud. /unsubscribe

After following this issue for a year, this is exactly the same reply I came here to make before unsubscribing from this issue.

Immich is amazing. You can point it at your nextcloud photos directory too, and just use it for read-only browsing of your files while still using nextcloud to manage the file organization and syncing.

@szaimen
Copy link

szaimen commented Jan 2, 2024

Actually there is some progress already in matiasdelellis/facerecognition-external-model#7 but I am waiting for @matiasdelellis to review the PR.

@szaimen szaimen mentioned this issue Jan 3, 2024
Merged
@szaimen
Copy link

szaimen commented Jan 3, 2024

After matiasdelellis/facerecognition-external-model#7 is merged, mainly only nextcloud/all-in-one#3999 is missing now :)

@szaimen
Copy link

szaimen commented Jan 10, 2024

mainly only nextcloud/all-in-one#3999 is missing now :)

This is now released with v7.10.0 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel and https://github.com/nextcloud/all-in-one/tree/main/community-containers/facerecognition

@luzfcb
Copy link

luzfcb commented Jan 10, 2024

@matiasdelellis @szaimen I started testing v7.10.0 Beta. Is there any way to improve performance of facerecognition-external-model?
For me, it analyzes 1 image every 16 seconds.
My server runs on Ryzen 5600G, 32 GB of RAM, and Nvidia GTX 1650, so I suppose it's more than enough to have a very good performance.

Well, this comment is completely offtopic of this issue. Is there somewhere we could continue discussions?

@szaimen
Copy link

szaimen commented Jan 10, 2024

Well, this comment is completely offtopic of this issue. Is there somewhere we could continue discussions?

I suppose here?
https://github.com/matiasdelellis/facerecognition/discussions

@matiasdelellis
Copy link
Contributor

Hi @luzfcb

My server runs on Ryzen 5600G, 32 GB of RAM, and Nvidia GTX 1650, so I suppose it's more than enough to have a very good performance.

Mainly it's great that you have an Nvidia GTX 1650 😬 , but the docker container is not compiled to use cuda. You should install CUDA on your host, and build the container to make use of it. 😉
It will be much faster. 😬
It shouldn't be complicated, but I can't help you because I never had a video card. 😞

On the other hand, you can reduce the size of temporary images on settings. By default this container is in 4Mpx, and with 2Mps you will get it will be 50% faster and you will get practically the same result.

I left 4mpx as the default because the larger the images, the more likely you are to find all your faces, but in practice, you don't get more than 10% new faces.

Finally no matter how long the process takes, it is designed to do the job progressively. We try not to take shortcuts but do better.

p.s: Not even Google gives you the results so quickly when you upload many images... 😅
Let's be pacient.. 😉

@szaimen
Copy link

szaimen commented Jan 10, 2024

Lets continue here: matiasdelellis/facerecognition#716

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

community-containers - add facerecognition nextcloud/all-in-one