Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#537 - change solr to no lock #552

Merged
merged 32 commits into from Jan 14, 2019
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
32672c5
#537 - change solr to no lock
twardnw Aug 9, 2018
fbf1989
move the solr data files and volume, leaving the schema in the contai…
Aug 29, 2018
e4984f3
Merge branch 'master' into 537-solr-lockfile
Aug 29, 2018
d6ea8b8
slight update to docs for new format.
Aug 30, 2018
13da91c
start framing tests.
Aug 30, 2018
3e29923
Solr relocation and foot shooting prevention script.
Sep 3, 2018
41c54dc
Merge branch 'master' into 537-solr-lockfile
Schnitzel Sep 9, 2018
280871c
Merge branch 'master' into 537-solr-lockfile
rtprio Sep 14, 2018
d70602f
Merge branch 'master' into 537-solr-lockfile
Sep 17, 2018
2ba5f78
clarify error message.
Sep 17, 2018
d3a4e65
move rather than copy; remove cleanup checks.
Sep 17, 2018
375c8c4
mismerge mishap.
Sep 17, 2018
168e104
use a single run in the solr docker example.
Sep 17, 2018
08df593
refactor a bit.
Sep 20, 2018
5fa6746
Merge remote-tracking branch 'origin/master' into 537-solr-lockfile
Schnitzel Nov 1, 2018
63a7061
use $() over ``.
Nov 7, 2018
31b0190
add some comments describing the process.
Nov 7, 2018
797ae20
prevent errors and wrap in an if.
Nov 7, 2018
cf9c132
remove exec in entrypoint.
Nov 7, 2018
6117f93
Merge branch 'master' into 537-solr-lockfile
rtprio Nov 15, 2018
3b2b8c3
Merge branch 'master' into 537-solr-lockfile
rtprio Dec 3, 2018
2515651
move cores from opt to var.
Dec 7, 2018
7a77787
move pvc conversion after the solr-port entrypoint
Dec 7, 2018
303295f
during testing, sometimes the datadir/data could not be removed; rena…
Dec 7, 2018
61287b1
detect if the datadir has been moved, do not try to do it again.
Dec 7, 2018
4445358
Merge branch 'master' into 537-solr-lockfile
Dec 7, 2018
051498a
Merge branch 'master' into 537-solr-lockfile
rtprio Dec 20, 2018
d731384
project id 19 is already used, increasing by one
Schnitzel Dec 26, 2018
3c9bc97
only create if not existing
Schnitzel Dec 28, 2018
9abc2a8
refactor solr-datadir entrypoing:
Schnitzel Dec 28, 2018
4ef0eb0
changing owner of /var/solr for local solr working
Schnitzel Dec 31, 2018
1364290
more refactoring:
Schnitzel Dec 31, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
16 changes: 13 additions & 3 deletions docs/using_lagoon/drupal/services/solr.md
Expand Up @@ -22,15 +22,25 @@ FROM amazeeio/solr:6.6

COPY .lagoon/solr /solr-conf/conf

CMD ["solr-precreate", "drupal", "/solr-conf"]
RUN precreate-core drupal /solr-conf

CMD ["solr-foreground"]
```

The goal is to have your solr configuration files exist at `/solr-conf/conf` in the image you are building.

## Multiple cores

To implement multiple cores, you will also need to ship your own solr schema as above, the only change needed is to the `CMD` of the Dockerfile, repeat the pattern of `precreate corename /solr-conf/ ;` for each core you require.
To implement multiple cores, you will also need to ship your own solr schema as above, the only change needed is to the `CMD` of the Dockerfile, repeat the pattern of `precreate-core corename /solr-conf/ ;` for each core you require.

```
CMD ["sh", "-c", "precreate-core drupal /solr-conf/ ; precreate-core core1 /solr-conf/ ; precreate-core core2 /solr-conf/ ; precreate-core core3 /solr-conf/ ; solr start -f"]

FROM amazeeio/solr:6.6-drupal

RUN precreate-core drupal-index1 /solr-conf && \
precreate-core drupal-index2 /solr-conf && \
precreate-core drupal-index3 /solr-conf

CMD ["solr-foreground"]

```
4 changes: 3 additions & 1 deletion images/solr-drupal/Dockerfile
Expand Up @@ -5,4 +5,6 @@ ARG SOLR_MAJ_MIN_VERSION

COPY solr${SOLR_MAJ_MIN_VERSION} /solr-conf

CMD ["solr-precreate", "drupal", "/solr-conf"]
RUN precreate-core drupal /solr-conf

CMD ["solr-foreground"]
4 changes: 2 additions & 2 deletions images/solr-drupal/solr5.5/conf/solrconfig.xml
Expand Up @@ -110,7 +110,7 @@
replication is in use, this should match the replication
configuration.
-->
<dataDir>${solr.data.dir:}</dataDir>
<dataDir>/var/solr/${solr.core.name}</dataDir>

<!-- The DirectoryFactory to use for indexes.

Expand Down Expand Up @@ -256,7 +256,7 @@
More details on the nuances of each LockFactory...
http://wiki.apache.org/lucene-java/AvailableLockFactories
-->
<lockType>${solr.lock.type:native}</lockType>
<lockType>${solr.lock.type:none}</lockType>

<!-- Expert: Controls how often Lucene loads terms into memory
Default is 128 and is likely good for most everyone.
Expand Down
4 changes: 2 additions & 2 deletions images/solr-drupal/solr6.6/conf/solrconfig.xml
Expand Up @@ -113,7 +113,7 @@
replication is in use, this should match the replication
configuration.
-->
<dataDir>${solr.data.dir:}</dataDir>
<dataDir>/var/solr/${solr.core.name}</dataDir>


<!-- The DirectoryFactory to use for indexes.
Expand Down Expand Up @@ -246,7 +246,7 @@
More details on the nuances of each LockFactory...
http://wiki.apache.org/lucene-java/AvailableLockFactories
-->
<lockType>${solr.lock.type:native}</lockType>
<lockType>${solr.lock.type:none}</lockType>

<!-- Commit Deletion Policy
Custom deletion policies can be specified here. The class must
Expand Down
24 changes: 24 additions & 0 deletions images/solr/00-solr-datadir.sh
@@ -0,0 +1,24 @@
#!/bin/sh
set -ex

# old volume looks like this
# $corename/data/-files-
# new volume looks like this
# $corename/-files-

for datadir in `ls -d /var/solr/*` ; do
rtprio marked this conversation as resolved.
Show resolved Hide resolved
corename=$(basename $datadir)
if [ -d ${datadir}/data ]; then
echo "${datadir}/data is in old format, moving to ${datadir}."
mv ${datadir}/data/* ${datadir}/
rm -Rf ${datadir}/data
fi

mkdir -p /opt/solr/server/solr/${corename}
rtprio marked this conversation as resolved.
Show resolved Hide resolved
cp -R ${datadir}/conf /opt/solr/server/solr/${corename}/
touch /opt/solr/server/solr/${corename}/core.properties
sed -ibak 's/<dataDir>.*/<dataDir>\/var\/solr\/${solr.core.name}<\/dataDir>/' /opt/solr/server/solr/${corename}/conf/solrconfig.xml

done

exec solr-foreground
rtprio marked this conversation as resolved.
Show resolved Hide resolved
14 changes: 10 additions & 4 deletions images/solr/Dockerfile
Expand Up @@ -23,17 +23,23 @@ ENV TMPDIR=/tmp \
# we need root for the fix-permissions to work
USER root

RUN fix-permissions /opt/solr/server/solr/mycores/ \
&& fix-permissions /opt/solr/server/logs
RUN mkdir /var/solr
RUN fix-permissions /var/solr \
&& fix-permissions /opt/solr/server/logs \
&& fix-permissions /opt/solr/server/solr


# solr really doesn't like to be run as root, so we define the default user agin
USER solr

COPY 00-solr-datadir.sh /lagoon/entrypoints/
COPY 10-solr-port.sh /lagoon/entrypoints/



# Define Volume so locally we get persistent cores
VOLUME /opt/solr/server/solr/mycores/
VOLUME /var/solr

ENTRYPOINT ["/sbin/tini", "--", "/lagoon/entrypoints.sh"]

CMD ["solr-precreate", "mycore"]
CMD ["solr-precreate", "mycore"]
21 changes: 21 additions & 0 deletions local-dev/api-data/01-populate-api-data.gql
Expand Up @@ -612,6 +612,27 @@ mutation PopulateApi {
id
}

CiSolr: addProject(
input: {
id: 19
name: "ci-solr"
customer: 3
openshift: 2
gitUrl: "ssh://git@192.168.99.1:2222/git/solr.git"
}
) {
id
}
CiSolrRocketChat: addNotificationToProject(
input: {
project: "ci-solr"
notificationType: ROCKETCHAT
notificationName: "amazeeio--lagoon-local-ci"
}
) {
id
}

#### Lagoon Kickstart Objects
# Customer with a private key that has access to the local-git server.
KickstartCustomer: addCustomer(
Expand Down
3 changes: 2 additions & 1 deletion local-dev/git/Dockerfile
Expand Up @@ -25,7 +25,8 @@ RUN mkdir -m 700 /git/.ssh && \
git --bare init /git/nginx.git && \
git --bare init /git/features.git && \
git --bare init /git/features-subfolder.git && \
git --bare init /git/elasticsearch.git
git --bare init /git/elasticsearch.git && \
git --bare init /git/solr.git
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that we add a new git repo, but is there also a new test for this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started; but then I realized I wasn't testing the feature that I was developing, only that solr came up and could index and search.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better than nothing :) how far are you? happy to get this change in and add the tests later


USER root

Expand Down