build: dockerize fuseki (dsp-30) #1653

subotic · 2020-06-10T08:01:56Z

add creating and publishing of own docker image: daschswiss/knora-fuseki
add running of tests with fuseki
remove GraphDB from Makefile
add repository create scripts (fuseki-init-knora-test.sh, etc.)
Upgrade integration test failing. Need to change scripts so they work with Fuseki
add automatic startup of containers when tests are running from from SBT.
SearchRouteV2R2RSpec

Postponed issues:

deleting the repository using the Fuseki Admin HTTP API does not work correctly. It needs to be deleted manually before running the fuseki-init-knora-test.sh script (DSP-432).

benjamingeer · 2020-06-12T17:25:53Z

should not return duplicate results when there are unbound variables in one or more UNION
branches

I can look at this next week.

Upgrade integration test failing. Need to change scripts so they work with fuseki

There are no scripts to change, everything is done in Knora, including support for upgrading a repository in Fuseki. It looks like the tests are failing in GitHub CI because git checkout fails. If there’s another error, let me know.

benjamingeer · 2020-06-12T18:19:58Z

OK I see, fuseki-empty-repository.sh and fuseki-upload-repository.sh (used only in the Makefile) are failing, maybe because Fuseki is running on a different port? Or not running in Tomcat?

subotic · 2020-06-12T21:59:17Z

OK I see, fuseki-empty-repository.sh and fuseki-upload-repository.sh (used only in the Makefile) are failing, maybe because Fuseki is running on a different port? Or not running in Tomcat?

I think it should be fixed now. The URL is a bit different because Fuseki is not running in Tomcat.

subotic · 2020-06-12T22:01:36Z

should not return duplicate results when there are unbound variables in one or more UNION
branches

I can look at this next week.

Thanks for taking a look. I don't really know what this test does.

subotic · 2020-06-12T22:19:53Z

Maybe we can talk on Monday. There are some limitations because removing a repository doesn't work over the Fuseki Admin HTTP API. The current Makefile plumbing is good enough for CI and for a front-end developer or research-IT that need to start the whole stack. But for a DSP-API developer it is not a nice experience. We should discuss how it should work for the developer. @SepidehAlassi mentioned https://www.testcontainers.org. By using this, the starting of the DB, Redis, and SIPI containers would happen automatically at the beginning of each test. As a developer, you would only need to start the tests, everything else would happen automagically.

subotic · 2020-06-16T14:44:37Z

@benjamingeer OK, you can work now on the one failing test. There is currently a problem with Github-CI.

To run the tests locally, you need to run make build-all-images before you run the tests from inside SBT so that the needed Docker images are built. You can, of course, also run make test if you like.

subotic · 2020-06-23T16:28:40Z

@benjamingeer I'm struggling to get the sipi integration tests working with testcontainers, because there is no way to mount the shared tmp folder.

Could we maybe rewrite /v1/resources to use convert_from_file.lua instead convert_from_path.lua? Or is there something I'm missing? I vaguely remember that this was the plan at some point?

benjamingeer · 2020-06-23T16:43:42Z

I think what you want is #1233, which is blocked by dasch-swiss/sipi#309.

subotic · 2020-06-23T21:01:03Z

I think what you want is #1233, which is blocked by dasch-swiss/sipi#309.

Yes, that's what I meant. Thanks!

…knora-api into wip/DSP-30-dockerize-fuseki

subotic · 2020-06-26T07:07:42Z

@benjamingeer It would be great if you could do a quick review. I would like to first merge this PR before I start working on the Sipi v3.0.0 PR.

benjamingeer · 2020-06-26T07:09:30Z

README.md


-First, open an [issue](https://github.com/dhlab-basel/Knora/issues) to describe your problem or idea. We may ask you to submit a [pull request](https://help.github.com/articles/about-pull-requests/) implementing the desired functionality.
+First, open an [issue](https://github.com/dhlab-basel/Knora/issues) to describe


Should be https://github.com/dasch-swiss/knora-api/issues now.

Thanks, fixed. I have found a few more old links and fixed those also.

benjamingeer · 2020-06-26T07:14:47Z

webapi/scripts/fuseki-init-knora-test.sh

+  fi
+}
+
+# delete-repository // delete dos not work correctly. need to delete database manually.


Why not send a SPARQL DROP ALL instead of deleting the repository?

Init is used to initialize the repository, i.e., create a new repository based on the configuration which can change. If we only empty the repository, then a new one cannot be created, because there is already one with that name.

The Admin API route for deleting the repository only deletes the TDB2 database but does not delete the Apache Lucene database, so when a new repository is created, an error message is returned. This is why the repository needs to be deleted by hand, or in our case through a special make target.

Could we just ignore those errors?

Never mind, I understand: you need to be able to recreate the repository with a different configuration.

benjamingeer · 2020-06-26T07:20:12Z

webapi/src/it/scala/org/knora/webapi/e2e/v1/KnoraSipiScriptsV1ITSpec.scala

+              * Send a GET request to Sipi, asking for the preview image.
+              * With testcontainers it is not possible to know the random port
+              * in advance, so the started sipi container cannot be configured
+              * correctly.


I don't understand. Does this mean that this test can't work with testcontainers? Does something need to be fixed here?

I have a workaround for this. In the tests, I rewrite the returned standard URL from Sipi to the correct one. We are probably overspecifying things in Sipi, which makes the whole thing a bit unflexible.

Could you clarify the comment? "the started sipi container cannot be configured correctly" makes it seem as if something is broken and needs to be fixed.

benjamingeer · 2020-06-26T07:22:48Z

webapi/src/main/scala/org/knora/webapi/app/ApplicationActor.scala

+              | |  _ \/ ___||  _ \       / \  |  _ \_ _|
+              | | | | \___ \| |_) |____ / _ \ | |_) | |
+              | | |_| |___) |  __/_____/ ___ \|  __/| |
+              | |____/|____/|_|       /_/   \_\_|  |___|


I'm confused about these names. I thought "Knora" and "DSP-API" were the same thing. Personally I think "Knora" is a better name than "DSP-API" (takes too long to say and is impossible to remember).

Knora is slowly phased out. The official name of the platform is DSP, hence DSP-API.

I think DSP is a terrible name. Do a Google search for "DSP" and see what you get. Can the whole team have a discussion about this?

Sure, we can do it next Tuesday after the Daily Scrum.

benjamingeer · 2020-06-26T07:24:10Z

webapi/src/main/scala/org/knora/webapi/store/triplestore/http/HttpTriplestoreConnector.scala

@@ -607,6 +600,65 @@ class HttpTriplestoreConnector extends Actor with ActorLogging with Instrumentat
        }
    }

+    /**
+      * Initialize the Jena Fuseki triplestore. Currently only works for 'knora-test'
+      * and 'knora-test-unit' repository names.


because here I'm using the two Fuseki configs that we have (knora-test and knora-test-unit). I guess I can make it configurable, but then maybe in another PR. I've created https://dasch.myjetbrains.com/youtrack/issue/DSP-434.

I'm just trying to understand the use case for this method. Does this method only need to be called for knora-test and knora-test-unit, or is this a limitation that needs to be dealt with?

This method, in my opinion, mainly makes sense for automated testing, to make life easier. To be used, the API needs to be started with a certain flag, which I personally would never use or recommend for production.

OK, then please just clarify that in the comment.

benjamingeer

OK, this is great, I'm really happy to see it finally working! Thank you!

subotic · 2020-06-26T10:08:08Z

Me too. Thanks for making it possible in the first place :-)

subotic · 2020-06-26T10:16:49Z

And thanks for the review :-)

subotic added 3 commits June 9, 2020 18:29

build: remove graphdb, add fuseki

9da83f7

build: add fuseki db init (ongoing)

198af81

build: add fuseki db init (ongoing)

97930ff

subotic self-assigned this Jun 10, 2020

subotic changed the title ~~Wip/dsp 30 dockerize fuseki~~ build: dockerize fuseki (dsp-30) Jun 10, 2020

subotic added 8 commits June 10, 2020 10:15

Merge branch 'develop' into wip/DSP-30-dockerize-fuseki

2ca6947

build: add fuseki db init (ongoing)

cc6e883

build: add fuseki db init (ongoing)

ad60816

build: add fuseki db init (ongoing)

ccfd9bb

build: add fuseki db init

017e699

Merge branch 'develop' into wip/DSP-30-dockerize-fuseki

0e16a71

build: remove graphdb specific settings from github-ci

0c18f07

build: fix test upgrade

c03ae5b

build: fix upgrade test

3208b24

subotic added 9 commits June 13, 2020 09:52

build: fix upgrade test

f98adf7

test: fix deprecation

cdddafd

build: add testcontainers (ongoing)

8c38dad

build: add deleting of database volumes when init-db-xyz targets are run

0e03f2e

build: add testcontainers (ongoing)

0822b20

build: add testcontainers (ongoing)

2f0b141

build: add testcontainers (ongoing)

994769e

build: add testcontainers (ongoing)

326d3dd

build: add testcontainers (ongoing)

2e7d550

docs: update readme

ecb233a

subotic added 9 commits June 18, 2020 17:48

build: fix github-ci sbt error {ongoing}

e4e4e47

Merge branch 'develop' into wip/DSP-30-dockerize-fuseki

6b85d8c

build: fix github-ci sbt error {ongoing}

2e13af5

build: change Fuseki config to TDB2

3568b06

build: add using adoptopenjdk for github-ci

30b4255

doc: make Fuseki more prominent

54e3d39

build: fix auto init config

0f70b6c

fix (sipi): use correct variable for existence checking

ff70108

Merge branch 'develop' into wip/DSP-30-dockerize-fuseki

2e28004

subotic added 3 commits June 24, 2020 09:30

test: fix integration tests (ongoing)

daf4594

Merge branch 'wip/DSP-30-dockerize-fuseki' of github.com:dasch-swiss/…

078931d

…knora-api into wip/DSP-30-dockerize-fuseki

tests: ignore tests requiring shared folder

70fbc6b

subotic requested a review from benjamingeer June 25, 2020 15:20

subotic marked this pull request as ready for review June 25, 2020 15:21

subotic added the enhancement improve existing code or new feature label Jun 25, 2020

Merge branch 'develop' into wip/DSP-30-dockerize-fuseki

2d281d1

benjamingeer suggested changes Jun 26, 2020

View reviewed changes

subotic added 3 commits June 26, 2020 09:44

docs: update readme

b2be4e0

test: update comment

eb5703f

doc: update comment

f117f7b

benjamingeer approved these changes Jun 26, 2020

View reviewed changes

subotic merged commit e4fd02c into develop Jun 26, 2020

subotic deleted the wip/DSP-30-dockerize-fuseki branch June 26, 2020 10:31

benjamingeer mentioned this pull request Dec 2, 2020

fix(sipi): Don't expect API v1 status code (DSP-1114) #1763

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: dockerize fuseki (dsp-30) #1653

build: dockerize fuseki (dsp-30) #1653

subotic commented Jun 10, 2020 •

edited

Loading

benjamingeer commented Jun 12, 2020

benjamingeer commented Jun 12, 2020

subotic commented Jun 12, 2020

subotic commented Jun 12, 2020

subotic commented Jun 12, 2020

subotic commented Jun 16, 2020

subotic commented Jun 23, 2020

benjamingeer commented Jun 23, 2020

subotic commented Jun 23, 2020

subotic commented Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020 •

edited

Loading

benjamingeer Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer Jun 26, 2020

subotic Jun 26, 2020

benjamingeer left a comment

subotic commented Jun 26, 2020

subotic commented Jun 26, 2020


		First, open an [issue](https://github.com/dhlab-basel/Knora/issues) to describe your problem or idea. We may ask you to submit a [pull request](https://help.github.com/articles/about-pull-requests/) implementing the desired functionality.
		First, open an [issue](https://github.com/dhlab-basel/Knora/issues) to describe

build: dockerize fuseki (dsp-30) #1653

build: dockerize fuseki (dsp-30) #1653

Conversation

subotic commented Jun 10, 2020 • edited Loading

benjamingeer commented Jun 12, 2020

benjamingeer commented Jun 12, 2020

subotic commented Jun 12, 2020

subotic commented Jun 12, 2020

subotic commented Jun 12, 2020

subotic commented Jun 16, 2020

subotic commented Jun 23, 2020

benjamingeer commented Jun 23, 2020

subotic commented Jun 23, 2020

subotic commented Jun 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subotic Jun 26, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjamingeer left a comment

Choose a reason for hiding this comment

subotic commented Jun 26, 2020

subotic commented Jun 26, 2020

subotic commented Jun 10, 2020 •

edited

Loading

subotic Jun 26, 2020 •

edited

Loading