Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: switch continuous integration system to Drone CI #4

Closed
wants to merge 2 commits into from
Closed

WIP: switch continuous integration system to Drone CI #4

wants to merge 2 commits into from

Conversation

nodiscc
Copy link
Owner

@nodiscc nodiscc commented Jan 4, 2021 •
edited
Loading

https://cloud.drone.io/nodiscc/Shaarli

This was referenced Jan 4, 2021
Closed
Closed
Closed
@nodiscc
Copy link
Owner Author

nodiscc commented Jan 6, 2021

I think using the php image as baseline might be the wrong choice. A lot of build dependencies have to be pulled to build php extensions, build times are quite long... The composer image might have been a better choice but I don't think you can target a specific PHP version with those images.

Also test failures

I will keep looking into it

/cc @ArthurHoaro @immanuelfodor

@immanuelfodor
Copy link

Hi nodiscc, awesome progress here, you're really a gem for Shaarli! 😍

The Shaarli org could have a drone-ci-base repo which could build the image needed for the CI (shaarli/drone-ci-base:master for example). This way, the CI could have everything prepackaged, and you could update the contents when needed with full control. It could also speed up CI runs as you don't need to download and install any build dependencies when building or testing Shaarli.

The repo could also rebuild that docker image with cron trigger periodically to get other package updates (eg. openssl) if that's a concern.

@ArthurHoaro
Copy link

Nice work!

Yes using the php image is nice for production image, because you can setup everything with the latest version, but they're long to build and we can't recompile everything for every tests.

I agree with Immanuel here. We create our PHP CI image with the dependencies we need and we should be fine. It shouldn't require too much support, only when Shaarli changes its dependencies or once a year when a new major PHP version is released.

Regarding the errors, they all seem related to the filesystem: writing file without permission to write does not raise exception. I'd guess it might be because tests are running as root; or it might be something specific with that image.

@immanuelfodor
Copy link

Shaarli is probably used on several platforms, as in regular x86 servers, RPis, etc. it might be sooner or later requested to provide Docker images to some of them.

Lately, I got a request to provide a multi-platform image for one of my projects built with Drone. As Drone's platform directive requires builder instances on those platforms, and I don't have ARM for example, I "cheated", and built with platform emulation using buildx. Here is the working pipeline with some caching to speed it up: https://github.com/immanuelfodor/kube-karp/blob/master/.drone.yml

Original ticket with some links to Drone docs: immanuelfodor/kube-karp#2

@ArthurHoaro
Copy link

@nodiscc I've initialized a Drone config file based on yours and jsonnet extension for the netscape bookmark parser. It currently hits the same permission issues as this PR.

https://github.com/ArthurHoaro/netscape-bookmark-parser/pull/1/files

@nodiscc nodiscc changed the title WIP: switch CI system to Drone CI WIP: switch continuous integration system to Drone CI Feb 23, 2021
@nodiscc
Copy link
Owner Author

nodiscc commented Feb 23, 2021 •
edited
Loading

The permission problem should be resolved. You need multiple things:

  • declare USER drone in the base image (and create the drone user + home directory) to prevent running tests as root
  • disable the builtin clone action and implement git clone in the pipeline (else the repo is cloned as root -> no write permissions during tests as drone)
  • add a src/ subdirectory in the base image with r/w access for drone. Only creating the drone user is not enough, if the src/ directory does not exist/have the correct permissions, Drone CI will create it as root (I couldn't find why and the build will fail.
  • todo: fix remaining build errors (missing jpeg support in php gd?)
  • todo: cache composer dependencies between builds http://plugins.drone.io/tags/cache/
  • todo: merge drone-ci-base Dockerfiles in the main Shaarli toolchain
  • todo: do not use autobuild features from hub.docker.com (long wait times, TD), let the drone pipeline build and push the images

@nodiscc
Copy link
Owner Author

nodiscc commented Mar 15, 2021

  • update phpunit configuration?
Warning:       Your XML configuration validates against a deprecated schema.
Suggestion:    Migrate your XML configuration using "--migrate-configuration"!

@nodiscc
Copy link
Owner Author

nodiscc commented Apr 3, 2021 •
edited
Loading

ok everything appears to be working except

@ArthurHoaro any idea?

Edit: https://stackoverflow.com/questions/50435632/iconv-return-incorrect-string-in-phpunit may be related

@ArthurHoaro
Copy link

gettext-related tests on php 7.4 (cloud.drone.io/nodiscc/Shaarli/125/1/1)

You should try to explicitly install related locales (de_DE her).

a thumbnailer test on php 7.3 cloud.drone.io/nodiscc/Shaarli/125/2/1

Error: Call to undefined function WebThumbnailer\Utils\imagejpeg()

I guess gd extension is not properly installed/configured/loaded.

@ArthurHoaro
Copy link

For extensions, I haved successfully used this in the past which made things way easier: https://github.com/mlocati/docker-php-extension-installer/

@nodiscc nodiscc self-assigned this Apr 6, 2021
nodiscc pushed a commit that referenced this pull request Apr 7, 2021
@ajabep
Fix a bug
2ea2c99 
When we try to access the atom feed and have no bookmarks, it raised the following exception :

```
Call to a member function reorder() on array /webroot/application/bookmark/BookmarkFileService.php:143
#0 /webroot/application/feed/FeedBuilder.php(106): Shaarli\Bookmark\BookmarkFileService->search(Array, 'public', false, false, true)
#1 /webroot/application/front/controller/visitor/FeedController.php(47): Shaarli\Feed\FeedBuilder->buildData('atom', Array)
#2 /webroot/application/front/controller/visitor/FeedController.php(20): Shaarli\Front\Controller\Visitor\FeedController->processRequest('atom', Object(Slim\Http\Request), Object(Slim\Http\Response))
#3 [internal function]: Shaarli\Front\Controller\Visitor\FeedController->atom(Object(Slim\Http\Request), Object(Slim\Http\Response), Array)
#4 /webroot/vendor/slim/slim/Slim/Handlers/Strategies/RequestResponse.php(40): call_user_func(Array, Object(Slim\Http\Request), Object(Slim\Http\Response), Array)
#5 /webroot/vendor/slim/slim/Slim/Route.php(281): Slim\Handlers\Strategies\RequestResponse->__invoke(Array, Object(Slim\Http\Request), Object(Slim\Http\Response), Array)
#6 /webroot/application/front/ShaarliMiddleware.php(55): Slim\Route->__invoke(Object(Slim\Http\Request), Object(Slim\Http\Response))
#7 [internal function]: Shaarli\Front\ShaarliMiddleware->__invoke(Object(Slim\Http\Request), Object(Slim\Http\Response), Object(Slim\Route))
#8 /webroot/vendor/slim/slim/Slim/DeferredCallable.php(57): call_user_func_array(Array, Array)
#9 [internal function]: Slim\DeferredCallable->__invoke(Object(Slim\Http\Request), Object(Slim\Http\Response), Object(Slim\Route))
#10 /webroot/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(70): call_user_func(Object(Slim\DeferredCallable), Object(Slim\Http\Request), Object(Slim\Http\Response), Object(Slim\Route))
#11 /webroot/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(117): Slim\Route->Slim\{closure}(Object(Slim\Http\Request), Object(Slim\Http\Response))
#12 /webroot/vendor/slim/slim/Slim/Route.php(268): Slim\Route->callMiddlewareStack(Object(Slim\Http\Request), Object(Slim\Http\Response))
#13 /webroot/vendor/slim/slim/Slim/App.php(503): Slim\Route->run(Object(Slim\Http\Request), Object(Slim\Http\Response))
#14 /webroot/vendor/slim/slim/Slim/MiddlewareAwareTrait.php(117): Slim\App->__invoke(Object(Slim\Http\Request), Object(Slim\Http\Response))
#15 /webroot/vendor/slim/slim/Slim/App.php(392): Slim\App->callMiddlewareStack(Object(Slim\Http\Request), Object(Slim\Http\Response))
#16 /webroot/vendor/slim/slim/Slim/App.php(297): Slim\App->process(Object(Slim\Http\Request), Object(Slim\Http\Response))
#17 /webroot/index.php(177): Slim\App->run(true)
#18 {main}
```
@nodiscc nodiscc force-pushed the drone-ci branch 2 times, most recently from 95603ed to 8ae1bd1 Compare April 13, 2021 18:43
@nodiscc
Copy link
Owner Author

nodiscc commented Apr 13, 2021

php/node pipelines are implemented https://cloud.drone.io/nodiscc/Shaarli/168. ~10s faster than previous Travis suite.

  • I tried to implement cache but it didn't appear to work, and it looks like there is some form of caching already (eg. composer packages are not re-downloaded).
  • There is some repetition, we can use the jsonnet pipeline syntax to make it more DRY but I think it loses in readability/maintainability
  • php8 tests are still failing
Your requirements could not be resolved to an installable set of packages.
Problem 1
 - Root composer.json requires pubsubhubbub/publisher dev-master -> satisfiable by pubsubhubbub/publisher[dev-master].
 - pubsubhubbub/publisher dev-master requires php ~5.4 || ~7.0 -> your php version (8.0; overridden via config.platform, actual: 8.0.3) does not satisfy that requirement.```
  • Docker autobuilds for base CI images are slow (long queue times, ~20 min). Base images will not change often so this may be ok. I'm not a fan of having to maintain a separate repo, so in the long run we can see if it's possible to use Drone to build/push these, using dockerfiles in the shaarli/Shaarli repo

@immanuelfodor
Copy link

I haven't tried this plugin yet but maybe the cache miss is related to one of these issues?

Btw 14-20s of build time is pretty neat anyways! 🏆

immanuelfodor
immanuelfodor suggested changes Apr 13, 2021
View reviewed changes
.drone.yml Outdated Show resolved Hide resolved
@nodiscc
Copy link
Owner Author

nodiscc commented Apr 15, 2021

hardcoded URL

fixed

https://cloud.drone.io/nodiscc/Shaarli/169/1/1

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires pubsubhubbub/publisher dev-master -> satisfiable by pubsubhubbub/publisher[dev-master].
    - pubsubhubbub/publisher dev-master requires php ~5.4 || ~7.0 -> your php version (8.0; overridden via config.platform, actual: 8.0.3) does not satisfy that requirement.

@ArthurHoaro pubsubhubbub/publisher explicitly requires php 5.6 or 7.0, I can add ignore-platform-req=php like phpunit already has, but I think pubsubhubbub plugin might be broken on PHP 8. I cannot test this at the moment. Let's add --ignore-platform-req=php and a notice somewhere that it's not guaranteed to work?

@ArthurHoaro
Copy link

I'll submit a PR there to fix that. In the meantime, if the UT are passing it's fine, composer configuration forces PHP 7.3.

@ArthurHoaro ArthurHoaro mentioned this pull request Apr 16, 2021
Merged
@nodiscc nodiscc force-pushed the drone-ci branch 3 times, most recently from 351157a to c23b4af Compare April 17, 2021 16:01
@nodiscc nodiscc force-pushed the drone-ci branch 5 times, most recently from d60b42b to d622b83 Compare April 17, 2021 16:24
@nodiscc
switch continuous integration system to Drone CI:
c3bb6c1 
- add .drone.yml, reimplement tasks from .travis.yml (php/node/python test suites)
- ref. shaarli#1649
- https://www.drone.io/enterprise/opensource/
- https://cloud.drone.io/shaarli/Shaarli/settings
- https://docs.drone.io/pipeline/docker/syntax/steps/
- https://docs.drone.io/pipeline/docker/syntax/cloning/
- https://docs.drone.io/pipeline/environment/reference/
- https://docs.drone.io/pipeline/configuration/
- https://docs.drone.io/pipeline/overview/
- https://docs.drone.io/pipeline/docker/syntax/volumes/
- https://docs.drone.io/pipeline/docker/syntax/volumes/temporary/
- disable builtin drone clone action since it only supports cloning as root (drone/drone-git#23, https://docs.drone.io/pipeline/docker/syntax/cloning/) + use shallow git clone
- php: use custom php base images with prebuilt PHP extensions and composer (speed up build process) (https://hub.docker.com/r/shaarli/drone-ci-base, https://www.github.com/shaarli/drone-ci-base, https://hub.docker.com/repository/docker/shaarli/drone-ci-base/builds)
- no need to port $PATH update command, the default node:10 image already has the correct $PATH
- don't force pull of latest images to speed up builds
- the default is to use a local/cached image when the image is not tagged :latest
- https://docs.drone.io/pipeline/docker/syntax/parallelism/
- https://docs.drone.io/pipeline/environment/reference/drone-git-http-url/
- ignore PHP version requirements for https://packagist.org/packages/pubsubhubbub/publisher, pending merge of pubsubhubbub/php-publisher#11
- closes shaarli#1649
@nodiscc
doc: update CI documentation
905e8d8 
- use cloud.drone.io badges for builds on the master branch
- stable/0.x badges will need to be updated when the next stable/0.x release is published
@nodiscc
Copy link
Owner Author

nodiscc commented Apr 17, 2021

pubsubhubbub/publisher dev-master requires php ~5.4 || ~7.0

There was a stray composer update without --ignore-platform-req=php in the pipeline...

Moved to shaarli#1741

@nodiscc nodiscc closed this Apr 17, 2021
@nodiscc nodiscc mentioned this pull request Apr 17, 2021
Merged
@nodiscc nodiscc mentioned this pull request May 17, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@immanuelfodor immanuelfodor immanuelfodor requested changes

Assignees

@nodiscc nodiscc

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@nodiscc @immanuelfodor @ArthurHoaro