Support FreeBSD

[jeffkaufman]

At one point mod_pagespeed supported FreeBSD, but it hasn't worked since ~1.2. It also has never worked with ngx_pagespeed, because ngx_pagespeed is more recent than that, but the additional effort of getting it working there should be small.

We've had various bugs opened for this over time. Some of them have information about working around build failures, but we need to track this in one place. Closing them all as duplicates, and linking them here:

• FreeBSD Support incubator-pagespeed-ngx#308
• FreeBSD support: gclient sync fails on "gyp: Undefined variable yasm_path in ....libjpeg.gyp" #1007
• FreeBSD support: Getting "'Z_NO_COMPRESSION' was not declared in this scope" if using system zlib #1009
• Compilation with libpng 1.5 fails #1010
• FreeBSD support: build fails on due to "'EXPECT_DEBUG_DEATH' was not declared in this scope" #1011
• FreeBSD Support #1055

[jeffkaufman]

Added to Accepted Feature Requests.

[mike-pt]

I actually submitted some if not most of those to the old google code repo, some I could get around others not so much, I'm also not a c/c++ expert but what I can do is start over fresh from the git repo and maybe create a GHI as problems appear? or do you simply prefer to track it all here?

[jeffkaufman]

@mike-pt We moved from google code to github because google code is going away, and separately I wanted to collect all the issues for FreeBSD support in one place. Now that we're on github, though, we can accept pull requests. So if you'd like to work on this, maybe figure out a code change and create a pull request?

[jeffkaufman]

For example, I think #1009 would make sense as a pull request on its own.

[mike-pt]

Yeah I know about google code, so only a matter of time, the exporter to GH is very handy though :)

Ok then, I'll give a new look into the GHI and see what I can fix myself and send PR, I'm sure I'll bump into something it won't be easy for me to fix but lets leave that for later :)

One thing I do remember is "gen/arch/" dirs are not created for freebsd, In my local version I jut copied over the linux one's where needed (I'll create a GHI about that later today, have to go now)

[jmarantz]

Beyond compilation patches, the strategic barriers to FreeBSD functionality are mostly around our reliance on posix shared memory, for 3 reasons:

• SharedMemory cache, which is not essential, as you can use a per-process LRU as an L1, or just don't use an L1. Dealing with this requires some coding around OS-based limitations, but is not difficult
• SharedMemory locks. This is essential for sane running of the system. We still have file-system-based locks in our code, but they perform poorly and we have switched the default to be shared-memory based, and no longer test file-based locks. However there is a new in-memory lock manager in our codebase, which will eliminate our need for shared-mem locks. But it requires some architectural level changes (a new pagespeed subprocess to run the lock manager and other central tasks + an RPC system to talk to it) to integrated into multi-process server-modules, and we are working on those now.
• SharedMemory statistics, which are currently needed to avoid exploding cpu usage due to concurrent image rewrites. While our stats will probably always be shared-memory based, our reliance on them for bounding the CPU usage of image rewrites should be eliminated once we have the central lock-manager process. Once that's the case, the FreeBSD system may remain hard to monitor (no stats) but it should be stable.

[morlovich]

More specifically, the issue is that we use pthread_mutexattr_setpshared.

memcached may be a good replacement for #1.

On Mon, Oct 26, 2015 at 10:24 AM, Joshua Marantz notifications@github.com
wrote:

Beyond compilation patches, the strategic barriers to FreeBSD
functionality are mostly around our reliance on posix shared memory, for 3
reasons:

SharedMemory cache, which is not essential, as you can use a
per-process LRU as an L1, or just don't use an L1. Dealing with this
requires some coding around OS-based limitations, but is not difficult

SharedMemory locks. This is essential for sane running of the system.
We still have file-system-based locks in our code, but they perform poorly
and we have switched the default to be shared-memory based, and no longer
test file-based locks. However there is a new in-memory lock manager in our
codebase, which will eliminate our need for shared-mem locks. But it
requires some architectural level changes (a new pagespeed subprocess to
run the lock manager and other central tasks + an RPC system to talk to it)
to integrated into multi-process server-modules, and we are working on
those now.

SharedMemory statistics, which are currently needed to avoid exploding
cpu usage due to concurrent image rewrites. While our stats will probably
always be shared-memory based, our reliance on them for bounding the CPU
usage of image rewrites should be eliminated once we have the central
lock-manager process. Once that's the case, the FreeBSD system may remain
hard to monitor (no stats) but it should be stable.

—
Reply to this email directly or view it on GitHub
#1073 (comment)
.

[mike-pt]

Nice to see this is getting some attention again :)

Does this mean you guys will be spending some time on it soon?

I'll be happy to test, other then trying to fix some compilation issue like I did before (and I had some help there too) I'm afraid I can't do much more...

But I would really love to see ngx_pagespeed support on FreeBSD.

[jeffkaufman]

Does this mean you guys will be spending some time on it soon?

Sorry to get your hopes up, but this isn't something we're currently planning on working on. We're just documenting our increasing reliance on posix shared memory in case someone else is thinking of working on this.

[mike-pt]

Well still better then nothing ;) tks

[wouteroostervld]

You could implement interprocess mutexes using umtx. (Golang is doing this).

Background:

• http://freebsd.1045724.n5.nabble.com/What-is-the-status-of-thread-process-shared-synchronization-td4224458.html
• https://lists.freebsd.org/pipermail/freebsd-hackers/2015-June/047835.html

[neveripe]

Sorry to annoy you, but are there any estimates on FreeBSD support?

Thanks in advance!

[jeffkaufman]

No one is currently working on this, or has told me they plan to work on it, so unless someone wants to pick this up and make this happen we're not going to see this anytime soon. Sorry!

[tarkhil]

Right now, compiling of mod_pagespeed failed.

In file included from third_party/chromium/src/base/third_party/dmg_fp/dtoa_wrapper.cc:46:
third_party/chromium/src/base/third_party/dmg_fp/dtoa.cc:1568:13: error: array subscript is of type 'char' [-Werror,-Wchar-subscripts]
if (!hexdig['0'])
^~~~
1 error generated.

gcc 4.2.1. attempt to build with gcc48 yielded lots of different errors. I guess I should try clang...

[tarkhil]

Nope, clang fails with the same error...

[morlovich]

Probably just change that to [48]?

gcc-4.2.1 isn't supported for recent versions, either way.

[oschaaf]

@mmokhi contributed a FreeBSD port of mod_pagespeed: https://svnweb.freebsd.org/changeset/ports/452011
X-Posting from mod-pagespeed-discuss:

The mod_pagespeed now exists as a port in official FreeBSD ports-tree (therefore pkg repository as well).
You simply can install the port by:
1- cd $PORTSDIR/www/mod_pagespeed; make; make install # Or use portmaster, poudriere or whatever you prefer in port-tools :D
OR
2- pkg install mod_pagespeed (make sure that your pkg-repo is on the latest then)

Thanks @mmokhi!

[bellwood]

===>  mod_pagespeed-1.12.34.2_1,1 is marked as broken: fails to configure.
*** Error code 1

11.1-RELEASE-p3

[mmokhi]

I noticed it recent days that it was marked by another committer as "broken" because it doesnt work with openssl of base because of nothaving pkgconf file in expected location (there are two openssl, one in ports and installable other is existing as default in base).
I'm working to fix this ASAP.

[jserviceorg]

at least it compiles for me flawless on latest 11.1-STABLE, further testing will follow

i removed the broken marking inside the makefile, and changed the apache version to 24, and added DEFAULT_VERSIONS+=ssl=libressl to /etc/make.conf

looks good so far

further testing will follow

[jserviceorg]

[==========] 172 tests from 19 test cases ran. (12656 ms total)
[ PASSED ] 172 tests.
root@webservices:/usr/ports/www/mod_pagespeed/work/mod_pagespeed-a5cfe15 #

pagespeed automatic test looks a bit different

[----------] Global test environment tear-down
[==========] 4331 tests from 396 test cases ran. (69500 ms total)
[ PASSED ] 4279 tests.
[ FAILED ] 52 tests, listed below:
[ FAILED ] PngOptimizerTest.ValidPngs
[ FAILED ] PngScanlineReaderRawTest.ValidPngsRow
[ FAILED ] PngScanlineReaderRawTest.ValidPngsEntire
[ FAILED ] PngScanlineWriterTest.DecodeGrayAlpha
[ FAILED ] ImageOptimizerTest.AllFormats
[ FAILED ] ImageOptimizerTest.PngToPng
[ FAILED ] ImageOptimizerTest.Resize
[ FAILED ] ImageOptimizerTest.NotResize
[ FAILED ] ImageConverterTest.OptimizePngOrConvertToJpeg
[ FAILED ] GifScanlineReaderRawTest.Interlaced
[ FAILED ] GifAnimationTest.ReadSingleFrameOpaque
[ FAILED ] GifAnimationTest.ReadSingleFrameTransparency
[ FAILED ] GifAnimationTest.ReadSingleFrameOpaqueFallingOffImage
[ FAILED ] GifAnimationTest.ReadSingleFrameOpaqueLargeFallingOffImage
[ FAILED ] GifAnimationTest.ReadSingleFrameTransparencyFallingOffImage
[ FAILED ] GifAnimationTest.ReadSingleFrameTransparencyFallingOffImageAtOrigin
[ FAILED ] GifAnimationTest.ReadSingleFrameOpaqueInZeroSizeImage
[ FAILED ] GifAnimationTest.ReadSingleFrameOpaqueInZeroSizeImageAtOrigin
[ FAILED ] GifAnimationTest.ReadSingleFrameInterlacedOpaque
[ FAILED ] GifAnimationTest.ReadSingleFrameInterlacedTransparency
[ FAILED ] GifAnimationTest.ReadSingleFrameInterlacedOpaqueFallingOffImage
[ FAILED ] GifAnimationTest.ReadSingleFrameInterlacedTransparencyFallingOffImage
[ FAILED ] GifAnimationTest.ReadSingleFrameDualColormapsOpaque
[ FAILED ] GifAnimationTest.ReadSingleFrameDualColormapsTransparency
[ FAILED ] GifAnimationTest.ReadSingleFrameDualColormapsOpaqueFallingOffImage
[ FAILED ] GifAnimationTest.ReadSingleFrameDualColormapsTransparencyFallingOffImage
[ FAILED ] GifAnimationTest.ReadSingleFrameDelayOpaque
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaque
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueFirstFallingOffImage
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueSecondFallingOffImage
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueFirstFallingOffImageAtOrigin
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueFirstFallingOffXImage
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueFirstFallingOffYImage
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueSecondFallingOffImageAtOrigin
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueNoDelay
[ FAILED ] GifAnimationTest.ReadMultipleFrameTransparency
[ FAILED ] GifAnimationTest.ReadMultipleFrameNoDelay2FrameOpaque
[ FAILED ] GifAnimationTest.ReadMultipleFrameTransparencySkipScanlines
[ FAILED ] GifAnimationTest.ReadMultipleFrameTransparencyMixInterlaced
[ FAILED ] GifAnimationTest.ReadMultipleFrameTransparencyMixColormaps
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueFallingOffImage
[ FAILED ] GifAnimationTest.ReadMultipleFrameTransparencyFallingOffImage
[ FAILED ] GifAnimationTest.ReadMultipleFrameOpaqueDisposal
[ FAILED ] GifAnimationTest.ReadMultipleFrameTransparencyLoopInfinite
[ FAILED ] GifAnimationTest.ReadMultipleFrameTransparencyNoDelayLoopInfinite
[ FAILED ] GifAnimationTest.ReadMultipleFrameTransparencyLoopThrice
[ FAILED ] StdioFileSystemTest.TestRecursivelyMakeDir_NoPermission
[ FAILED ] StdioFileSystemTest.TestLockBumping
[ FAILED ] ImageTest.PngAlphaToWebpLaTest
[ FAILED ] ImageTest.DrawImageDetails
[ FAILED ] ImageRewriteTest.ResizeTransparentImage
[ FAILED ] ImageRewriteTest.DebugMessageImageInfo

52 FAILED TESTS

[mmokhi]

@jserviceorg yeah it's supposed to work with libressl and openssl from ports (as opposite of base).
I'm in process of marking it only broken for openssl of base, next step is to make it work (configure) even with openssl of the base.

[denigmus]

Still no plans for ngx_pagespeed support FreeBSD?

[oschaaf]

@denigmus As far as I know no-one currently is looking into ngx_pagespeed on FreeBSD.

Having said that, @mmokhi's tackled nearly all of the technical platform challenges, so with that as a starting point getting ngx_pagespeed operational on FreeBSD should probably be relatively easy.

[denigmus]

@mmokhi
Could you please give a step by step guide what should be done to compile nginx with ngx_pagespeed support on 11.1?
I only see port /usr/ports/www/mod_pagespeed, which is described as for Apache
ap24-mod_pagespeed-1.12.34.2_9,1

[mmokhi]

On FreeBSD there is a Nginx port in the port-tree which is very straightforward to build (make [config]; make install;) and even easy to extend.
The thing that slowed me to add nginx pagespeed to it, was this part from the official nginx-pagespeed build guide 👇

psol_url=https://dl.google.com/dl/page-speed/psol/${NPS_RELEASE_NUMBER}.tar.gz
[ -e scripts/format_binary_url.sh ] && psol_url=$(scripts/format_binary_url.sh PSOL_BINARY_URL)
wget ${psol_url}

There are two problems with this (that I'm trying to solve 😃 😅 )

1. We can't assume a Linux shared-object to work just straightforward in FreeBSD
2. It's not nice to copy/use binaries in port-tree IMO, during build

I guess the PSOL being copied is the shared-object being produced while building apache-pagespeed...
If so we just can make the nginx depending on the apache's mod_pagepseed in FreeBSD and then copy the file and make the nginx-pagespeed built...

It just depends on if my guess is right 😅

[StormRU]

Hi!
I have installed mod_pagespeed. I am running Apache 2.4.33
Google's doc says that i should use mod_pagespeed_ap24.so, but we only have mod_pagespeed.so in this install. Is it still usable with Apache 2.4.33 and newer?

I have changed config to use mod_pagespeed.so

<IfVersion >= 2.4.2>
  <IfModule !access_compat_module>
    LoadModule access_compat_module /usr/local/libexec/apache24/mod_access_compat.so
  </IfModule>
  LoadModule pagespeed_module /usr/local/libexec/apache24/mod_pagespeed.so
</IfVersion>

[mike-pt]

I seems mod_pagespeed on freebsd is no longer broken thanks to you @mmokhi?

Do you have step on how to build nginx+pagespeed on freebsd? Its been a while since I tired this :D

[denigmus]

@mike-pt

I seems mod_pagespeed on freebsd is no longer broken thanks to you @mmokhi?

Why you decided so?

[mike-pt]

I didn't decide anything :D but the reason I said that is cause the port is in fact no longer marked broken https://svnweb.freebsd.org/ports?view=revision&revision=490804

Since that was one of the main problems I wonder if it should now be possible to build ngx_pagespeed, but Im not sure about what steps to take to do it.

[mmokhi]

@mike-pt
Hey, Sorry for delay in my response,
The steps I wrote here are pretty much everything we have to do.
On nginx-port part it's not a problem at all since it already has a lot of big and small plugins. The tricky part would be building PSOL object thing.
As far as I tested we can't use the already-built ones since they are built for Linux.

It shouldn't be super complicated but needs time. If you wanna start it I will be very happy if I can help you on any part.

[mike-pt]

Ah yeah back when the code wasn't even on github I played with that and even submitted a few patches uptream, but my C knowledge wasn't good enough to to more then check a few ifdefs and adapt for freebsd, the fact that this is now all on github also makes things easier so maybe I'll take a look at that....

[mmokhi]

Good then,
I'd be happy to help with C part etc if you had any questions or needed any help.

[sharmashivanand]

Apparently the file "mod_pagespeed_ap24.so" is missing from the install. I tried the pkg as well as the ports build. So currently this is broken.

I'm on freebsd-version: 12.0-RELEASE-p5 and Apache/2.4.39 (FreeBSD) OpenSSL/1.1.1a-freebsd configured with event MPM.

Hi!
I have installed mod_pagespeed. I am running Apache 2.4.33
Google's doc says that i should use mod_pagespeed_ap24.so, but we only have mod_pagespeed.so in this install. Is it still usable with Apache 2.4.33 and newer?

I have changed config to use mod_pagespeed.so

<IfVersion >= 2.4.2>
  <IfModule !access_compat_module>
    LoadModule access_compat_module /usr/local/libexec/apache24/mod_access_compat.so
  </IfModule>
  LoadModule pagespeed_module /usr/local/libexec/apache24/mod_pagespeed.so
</IfVersion>