Collection of issues to run Lucida

[k0105]

Here are some problems I have encountered so far:

Build process still quite brittle and not working under Ubuntu 16.04 LTS

While we're fine as long as we only use Lucida via Kubernetes, code modifications generally lead to problems, cmp. #116

ASR Insufficient

Firefox not supported and open source ASR clearly too weak for any real application (need proprietary IBM, Google, Microsoft or Nuance STT)

Lucida is prone to conflicts on existing settings

While Lucida builds on clean systems for me I ran into serious issues when trying to deploy it on productive systems with existing dependencies. Both times I've tried this Lucida failed to compile, one time it destroyed dependencies for another application so that afterwards both wouldn't run anymore.

Need Service Registry, optional: launch containers on demand

Internal containers cannot be shared via public service registry, so we need a local one

[Optional: With an increasing number of services Lucida becomes difficult to handle with lots of idle containers - we might want to alleviate this by launching containers only when they are needed. The command center needs to support this and list new services.]

Some dependencies hidden

/home/lucida_data
But also /usr/local/lucida/lucida ($LUCIDAROOT) and optional Wikipedia content /usr/local/lucida/lucida/questionanswering/OpenEphyra/db/wiki_indri_index
Those should be neatly collected in one data directory within Lucida and the user's home directory - another home directory and some system directory are too difficult to find. Not sure they are needed - if I setup Lucida without them, it still works.

Compilation runs so long it forgets sudo credentials

sudo will only remember its credentials for 15 minutes, but some steps in the compilation take longer (tested on an i7 with 64GB of RAM), so the user will be prompted for sudo credentials multiple times, which is annoying since the user cannot just walk away and let Lucida compile for an hour or two. It was also requested in issues like #143

We can either remove sudo from inside the script altogether (and use sudo -u for parts where things have to be run in userland) or we disable the timeout via
echo "\nDefaults timestamp_timeout=-1">>/etc/sudoers
and then sed -i "/Defaults timestamp_timeout=-1/d" /etc/sudoers to activate it again.
Those commands should be run with visudo instead of sudo, since this checks for syntax errors, which would break sudo entirely. We might also think about bundling commands that need superuser privileges in one script and build everything else in userland - that feels cleaner to me than having root access all over the place.

[DONE] Repository for Installation only could be reduced in size by a factor of 6000 and installation from scratch on fresh system is not supported

See https://github.com/k0105/mini-lucida
Lucida does not set up Docker, Docker Compose and its data directories. I have already written scripts that do, so Lucida can be deployed on clean systems.
Also, the full repository is much larger than what is needed to only run Lucida - a mini repository that suffices to run it is 6000x smaller.

[DONE] Generic Backend and outside orchestration

Generic backend now works in Lucida.
See https://github.com/k0105/lucidagenericbackend
This would help new developers to get started more quickly in adding new services.
I have a working server / client Thrift pair that uses the Lucida interface (server in Java, client in Python), but I still need to plug it into Lucida.
[David wanted Node-RED orchestration, which is partially incompatible with the proprietary Thrift interface, so we should discuss how to enable that as well.]

Make Lucida more platform independent

[DONE] Lucida currently does not compile on Ubuntu 14.04

I changed a few parts and now it compiles under Ubuntu 14.04.5 at least.
My changed code is here: https://github.com/k0105/lucida
The only commands I used on a fresh Ubuntu 14.04.5 just now were:

sudo apt install git make
git clone https://github.com/k0105/lucida
cd lucida
make local

[TODO] Lucida currently does not compile on Ubuntu 16.04

My attempt is here: https://github.com/k0105/lucida_experimental but it fails regarding fbthrift (possibly related to folly and protobuf). As far as I know Lucida has never worked on 16.04 - at least people are having massive issues with it. Still, as the current long term support release it should be supported.

[Optional] Does not run on Windows via minikube

Now that Lucida runs on macOS it would be neat to have Windows support as well. Unfortunately, an initial attempt failed:

mkdir $HOME\Documents\Lucida
$client = New-Object System.Net.WebClient
$client.DownloadFile("https://download.docker.com/win/stable/DockerToolbox.exe", "$HOME\Documents\Lucida\dockertoolbox.exe")
& "$HOME\Documents\Lucida\dockertoolbox.exe"
$client.DownloadFile("http://storage.googleapis.com/kubernetes-release/release/v1.4.0/bin/windows/amd64/kubectl.exe", "$HOME\Documents\Lucida\kubectl.exe")
$client.DownloadFile("https://storage.googleapis.com/minikube/releases/v0.16.0/minikube-windows-amd64.exe", "$HOME\Documents\Lucida\minikube.exe")
# Copy to C:\kubectl.exe and C:\minikube.exe
$files = @(Get-ChildItem $HOME\Documents\Lucida\mini-lucida\*-service.yaml)
foreach ($file in $files) {.\kubectl create -f $file}
$files = @(Get-ChildItem $HOME\Documents\Lucida\mini-lucida\*-controller.yaml)
foreach ($file in $files) {.\kubectl create -f $file}

Windows 7 runs out of memory during container start
Windows 10 gives “Unable to connect to the server: dial tcp [::1]:8080: connectex: No connection could be made because the target machine actively refused it.” even with deactivated firewalls (Windows and Comodo)

[sergani]

Hello, I'm not sure if this is related to my setup (Ubuntu 14.04 LTS Server), but opencv fails to build.
The install_opencv script fails in two lines:

check_opencv.cpp:2:33: fatal error: opencv2/core/core.hpp: No such file or directory
#include "opencv2/core/core.hpp"

Any thoughts?

[sergani]

Hi there thanks for the feedback. Never mind the mentioned issue, it was related to an unclean attempt to build. I've moved the project to a larger Ubuntu 14.04 vm and opencv built successfully. Working on the rest of the project, but it looks OK so far. Thanks //M On Feb 20, 2017 4:10 PM, "Κοῖος" <notifications@github.com> wrote: Interesting, that didn't happen for me. Sorry - I don't know Lucida too well and also have no idea. If I had designed Lucida, I would have pulled OpenCV as a binary dependency. While I really like using OpenCV, it is oftentimes difficult to build. I recently built version 3 for Caffe (I think) and that took me 3 hours or so. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#151 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQFlnw-xCLS7QiWKL3nhMUPpV9vdNE7Uks5reZ7qgaJpZM4L89XD> .