Build nouveau package #125
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
janl
reviewed
Dec 12, 2023
debian/control.in
Outdated
| lsb-base, | ||
| procps, | ||
| ${misc:Depends} | ||
| Description: Nouveau is a modern replacement for dreyfus/clouseau |
There was a problem hiding this comment.
maybe something high-level front, along the lines of ~“Nouveau adds Lucene-style indexing and searching to Apache CouchDB. …” then the rest?
debian/couchdb-nouveau.templates
Outdated
| Template: couchdb-nouveau/postrm_remove_indexes | ||
| Type: boolean | ||
| Default: false | ||
| _Description: Remove all CouchDB Nouveau indexes? |
There was a problem hiding this comment.
Suggested change
| _Description: Remove all CouchDB Nouveau indexes? | |
| _Description: Remove all CouchDB Nouveau indexe data? |
maybe this makes it more clear that this is something one cares about?
debian/couchdb-nouveau.templates
Outdated
| Type: boolean | ||
| Default: false | ||
| _Description: Remove all CouchDB Nouveau indexes? | ||
| The /var/opt/couchdb-nouveau/ directory containing all CouchDB databases will |
There was a problem hiding this comment.
Suggested change
| The /var/opt/couchdb-nouveau/ directory containing all CouchDB databases will | |
| The /var/opt/couchdb-nouveau/ directory containing all Nouveau indexes will |
copypasta mistake?
rnewson
force-pushed
the
nouveau
branch
3 times, most recently
from
10999e5
to
18dfa12
Compare
nickva
reviewed
Dec 12, 2023
rnewson
force-pushed
the
nouveau
branch
5 times, most recently
from
92d11ec
to
ea538d5
Compare
rnewson
force-pushed
the
nouveau
branch
7 times, most recently
from
9ad1f62
to
415ccd8
Compare
rnewson
force-pushed
the
nouveau
branch
7 times, most recently
from
e7669df
to
afdd218
Compare
nickva
reviewed
Mar 23, 2024
There was a problem hiding this comment.
I built the package for debian buster and ubuntu bionic
(pr/125)$ tree pkgs
pkgs
├── couch
│ ├── debian-buster
│ │ ├── couchdb-dbgsym_3.3.3-b964a84~buster_amd64.deb
│ │ ├── couchdb-nouveau_3.3.3-b964a84~buster_amd64.deb
│ │ ├── couchdb_3.3.3-b964a84~buster_amd64.buildinfo
│ │ ├── couchdb_3.3.3-b964a84~buster_amd64.changes
│ │ └── couchdb_3.3.3-b964a84~buster_amd64.deb
│ └── ubuntu-bionic
│ ├── couchdb-dbgsym_3.3.3-b964a84~bionic_amd64.ddeb
│ ├── couchdb-nouveau_3.3.3-b964a84~bionic_amd64.deb
│ ├── couchdb_3.3.3-b964a84~bionic_amd64.buildinfo
│ ├── couchdb_3.3.3-b964a84~bionic_amd64.changes
│ └── couchdb_3.3.3-b964a84~bionic_amd64.deb
On buster I first installed the base package. I noticed that it enabled nouveau even though the nouveau package wasn't installed yet.
[nouveau]
enable = true
url = http://127.0.0.1:5987
Should that disabled until we install the nouveau package? If it's tricky to do it, it's probably fine as is.
Then installed nouveau package. Noticed java package complained about installing certificate for some reason:
Unpacking openjdk-11-jre-headless:amd64 (11.0.22+7-1~deb10u1) ...
Selecting previously unselected package default-jre-headless.
Preparing to unpack .../default-jre-headless_2%3a1.11-71_amd64.deb ...
Unpacking default-jre-headless (2:1.11-71) ...
Setting up java-common (0.71) ...
Setting up ca-certificates-java (20190405) ...
head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory
Exception in thread "main" java.lang.InternalError: Error loading java.security file
at java.base/java.security.Security.initialize(Security.java:94)
at java.base/java.security.Security$1.run(Security.java:79)
at java.base/java.security.Security$1.run(Security.java:77)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.Security.<clinit>(Security.java:77)
at java.base/sun.security.jca.ProviderList.<init>(ProviderList.java:176)
at java.base/sun.security.jca.ProviderList$2.run(ProviderList.java:94)
at java.base/sun.security.jca.ProviderList$2.run(ProviderList.java:92)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.jca.ProviderList.fromSecurityProperties(ProviderList.java:91)
at java.base/sun.security.jca.Providers.<clinit>(Providers.java:54)
at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:156)
at java.base/java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:193)
at org.debian.security.KeyStoreHandler.<init>(KeyStoreHandler.java:50)
at org.debian.security.UpdateCertificates.<init>(UpdateCertificates.java:65)
at org.debian.security.UpdateCertificates.main(UpdateCertificates.java:51)
dpkg: error processing package ca-certificates-java (--configure):
installed ca-certificates-java package post-installation script subprocess returned error exit status 1
I tried reinstalling nouveau and the same error happened:
$ sudo apt install ./couchdb-nouveau_3.3.3-b964a84~buster_amd64.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'couchdb-nouveau' instead of './couchdb-nouveau_3.3.3-b964a84~buster_amd64.deb'
couchdb-nouveau is already the newest version (3.3.3-b964a84~buster).
0 upgraded, 0 newly installed, 0 to remove and 288 not upgraded.
2 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up ca-certificates-java (20190405) ...
head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory
Exception in thread "main" java.lang.InternalError: Error loading java.security file
at java.base/java.security.Security.initialize(Security.java:94)
at java.base/java.security.Security$1.run(Security.java:79)
at java.base/java.security.Security$1.run(Security.java:77)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.Security.<clinit>(Security.java:77)
at java.base/sun.security.jca.ProviderList.<init>(ProviderList.java:176)
at java.base/sun.security.jca.ProviderList$2.run(ProviderList.java:94)
at java.base/sun.security.jca.ProviderList$2.run(ProviderList.java:92)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.jca.ProviderList.fromSecurityProperties(ProviderList.java:91)
at java.base/sun.security.jca.Providers.<clinit>(Providers.java:54)
at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:156)
at java.base/java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:193)
at org.debian.security.KeyStoreHandler.<init>(KeyStoreHandler.java:50)
at org.debian.security.UpdateCertificates.<init>(UpdateCertificates.java:65)
at org.debian.security.UpdateCertificates.main(UpdateCertificates.java:51)
dpkg: error processing package ca-certificates-java (--configure):
installed ca-certificates-java package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of openjdk-11-jre-headless:amd64:
openjdk-11-jre-headless:amd64 depends on ca-certificates-java (>= 20190405~); however:
Package ca-certificates-java is not configured yet.
dpkg: error processing package openjdk-11-jre-headless:amd64 (--configure):
dependency problems - leaving unconfigured
Processing triggers for ca-certificates (20200601~deb10u2) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Exception in thread "main" java.lang.InternalError: Error loading java.security file
at java.base/java.security.Security.initialize(Security.java:94)
at java.base/java.security.Security$1.run(Security.java:79)
at java.base/java.security.Security$1.run(Security.java:77)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.Security.<clinit>(Security.java:77)
at java.base/sun.security.jca.ProviderList.<init>(ProviderList.java:176)
at java.base/sun.security.jca.ProviderList$2.run(ProviderList.java:94)
at java.base/sun.security.jca.ProviderList$2.run(ProviderList.java:92)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.jca.ProviderList.fromSecurityProperties(ProviderList.java:91)
at java.base/sun.security.jca.Providers.<clinit>(Providers.java:54)
at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:156)
at java.base/java.security.cert.CertificateFactory.getInstance(CertificateFactory.java:193)
at org.debian.security.KeyStoreHandler.<init>(KeyStoreHandler.java:50)
at org.debian.security.UpdateCertificates.<init>(UpdateCertificates.java:65)
at org.debian.security.UpdateCertificates.main(UpdateCertificates.java:51)
E: /etc/ca-certificates/update.d/jks-keystore exited with code 1.
done.
Errors were encountered while processing:
ca-certificates-java
openjdk-11-jre-headless:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)
I purged all the java packages, re-installed then installed couchdb-nouveau and it seemed to have worked.
The service came up automatically, which was very nice!, and I could make a simple query.
I purged couchdb-nouveau and it promoted if I wanted the data removed, responded "yes" and it succeeded.
After purging couchdb-nouveau, I noticed nouveau remained enabled and the server after restart was writing lots indicating it was looking for the nouveau service and returning a 503 "Service not available" if still made the query.
[warning] 2024-03-23T05:32:05.656090Z couchdb@127.0.0.1 <0.563.0> 86d88c9d84 search unavailable, retrying (11 of 20)
[warning] 2024-03-23T05:32:06.158340Z couchdb@127.0.0.1 <0.563.0> 86d88c9d84 search unavailable, retrying (12 of 20)
[warning] 2024-03-23T05:32:06.663945Z couchdb@127.0.0.1 <0.563.0> 86d88c9d84 search unavailable, retrying (13 of 20)
[warning] 2024-03-23T05:32:07.169055Z couchdb@127.0.0.1 <0.563.0> 86d88c9d84 search unavailable, retrying (14 of 20)
[warning] 2024-03-23T05:32:07.673765Z couchdb@127.0.0.1 <0.563.0> 86d88c9d84 search unavailable, retrying (15 of 20)
|
I can't repro the java cert issue but https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/1998065 sounds related. it's not a bug in the nouveau package, at least. |
rnewson
force-pushed
the
nouveau
branch
2 times, most recently
from
04bd3c6
to
b17642e
Compare
6 tasks
|
I tried rebasing on the latest main using Erlang 25 as the default and noticed a spotlessJava error when building packages: |
|
@nickva cannot reproduce that. I ran couchdb-pkg locally with a tarball of 4c52700, but it passes. the spotless error is a claim that Lucene9Index.java doesn't declare a class, which it clearly does, and it references line 2, which is a comment. So I suspect something else is going on, perhaps the file has some other content than expected (perhaps conflict markers?). Let's see if this is repeatable. if so, we'll need to tweak makefile to print this file to the build output |
|
Thanks @rnewson, perhaps there is something flaky in my setup. One step I forgot to mention is I rebased nouveau on main branch to make sure we're using OTP 25 in couchdb-pkg. If you get chance, could you rebase on main. I try to re-run everything again. With a rebased nouveau branch, latest main commit UPDATE: x86_64 builds all worked when rebased on main (OTP 25) |
|
I noticed the failure on ppc64el architecture. I'll try exclude it. |
|
Thanks for updating the PR @rnewson. I gave it a try and rebuilt all the package with this PR (rebased on main) One of the arches, ppc64le failed. Made an issue: #140 so I disabled it. Taking the approach that given a choice, we'd rather have nouveau support vs ppc64le for now. I tried installing just the couchdb package on a bullseye VM but still see the The dist tarball I used doesn't have it enabled, it's |
|
can you read the postinst inside those files and check for this code? |
|
The couchdb-nouveau package has it in the postinst script. Extracted the bits with |
|
It's probably this that's doing it: Lines 24 to 29 in 365d07c |
|
I am thinking of reverting that commit or fixing it such that we build all the artifacts ( Maybe another approach is that it's "enabled" as it is currently but we have an runtime availability state, which pings with nouveau service and keeps a runtime state of up/not_up... |
rnewson
force-pushed
the
nouveau
branch
2 times, most recently
from
0d9415a
to
aadcf31
Compare
|
updated in light of apache/couchdb#5105 so that default.ini never gets the |
|
thank you, I'll re-build the dist tarball from the latest main, rebuild the packages and re-test |
|
I created the base CouchDB and Nouveau packages for Bullseye with the latest changes. Main package installed fine. Nouveau feature was disabled as expected. With the running CouchDB server, installed the Nouveau package. Install went well, it asked for an option to enable it. I check it as "yes". I noticed the new .ini file was created in the right place and had the expected content. Enabling Nouveau didn't take effect though after Then I tried running a simple script which uses the _nouveau_analyze endpoint and runs a small query: #!/bin/bash
set -x
http -b post $DB/_nouveau_analyze analyzer=standard text=for@bar@baz
http -b delete $DB/ndb
http -b put $DB/ndb
http post $DB/ndb/_bulk_docs docs:='[{"_id":"1","a":"xxx"},{"_id":"2","a":"xxx"},{"_id":"3","a":"zzz"},{"_id":"4","a":"xxx"}]'
http -b put $DB/ndb/_design/nd1 nouveau:='{"idx1": {"index":"function(doc){index(\"string\",\"a\",doc.a,{\"store\":true})}"}}'
http -b post $DB/ndb/_design/nd1/_nouveau/idx1 q="a:xxx" limit:='2'$DB is the server URL DB=http://admin:pass@localhost:5984Everything ran ok until the last query Looking in CouchDB logs I noticed it was full of: Debugging further indicated that the base CouchDB deb package didn't install the server .js files. It was trying to run:
But the installed files for base CouchDB package doesn't have that path. Not sure why the file is not there, will have to debug further, later. So far it's as far as I got tonight. |
I think that’s because the list of ini files is loaded during the start of CouchDB and is “static” afterwards. Maybe we have to re-read those files on reload? |
Iirc I had something similar on Windows but I hadn’t time to dive in. My guess is some configure stuff with SM/Quickjs … |
|
oh that's icky, that we don't notice new .ini files except at startup. |
This was referenced Jul 3, 2024
nickva
approved these changes
Jul 4, 2024
There was a problem hiding this comment.
+1
Nicely done!
I have fixes for the two issues noticed during the review, they are general issues and not related to this PR:
- Make sure config reload finds new .ini files in .d directories couchdb#5114 : ini files in config directories
- Ensure js files end up in the correct release directory couchdb#5115 : missing js files in the package
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.