Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
add Solaris FEN support #263
This pull request fixes #12.
I'm an employee of Joyent, Inc., and have completed the Google corporate CLA.
What does this pull request do?
This pull request adds fsnotify support for Solaris and illumos via Solaris FEN. It is heavily based on #196, authored by @gfrey. As such, and with his approval, I have added @gfrey in the
One major revision to @gfrey's work concerns liveness when the watcher is closed when there is an unconsumed event in the channel - the old pull request would sometimes hang on the TestWatcherClose test. The new concurrency model is borrowed from fsnotify's inotify implementation.
Another major revision concerns proper translation of FEN events to fsnotify events, in particular the
Where should the reviewer start?
These pages are useful as an introduction to FEN:
The illumos man pages for port_create and related functions are also helpful for understanding how to interface with FEN.
The discussion on #196 is also worth reading. Some points I'll address here:
How should this be manually tested?
I ran all tests on a SmartOS host. On SmartOS, Go can be installed from Joyent's pkgsrc repository. fsnotify can then be built and tested as normal.
Looks like macOS builds are still failing even with the fix.
@nathany Joyent would be happy to offer a Triton container on the Joyent Public Cloud in which to build and test fsnotify. Under-the-hood this is a SmartOS zone that we'd administer and give you access to. I could assist with installing Go and setting up the build. You could then integrate this with the CI system of your choice. Please let me know if you're interested or have questions - feel free to email email@example.com.
@isaacdavis A CI solution would be great, though I can't say I have any idea how to integrate a Triton container with a hosted solution like Travis CI.
Also, before adding Solaris support, I want to be sure there is a point person (or people would be better) for Solaris support that are handling any issues. /cc @gfrey
We also want to be sure that Solaris support is documented, especially if there are behaviour differences between operating systems that users need to know about.
One other note: at GopherCon we have been discussing standalone packages for each OS that fsnotify relies on. This would likely look like
I've discovered a bug in the current implementation under review:
This bug does not manifest in the test suite, as there is not enough memory churn to trigger an overwrite of the
I'll have a fix committed by tomorrow (2018-09-06) at noon PST.
nathany left a comment
Thanks for your work on this Isaac.
For this review, please keep in mind that I'm not familiar with FEN, nor have I exercised this code or even ensured that it compiles (I don't have a Solaris environment, nor am I personally planning to set one up).
Structurally, I would prefer if the cgo code were more isolated as we have done for fsevents, but I'm not sure if that can even be done to the same extent.
It appears that FEN requires that a malloced file_obj_t remain allocated for the lifetime of the watch, is that correct?
@isaacdavis Apologies for the delay in reviewing your work. Here are some considerations before merging this.
I'm not personally willing to take on the responsibility of supporting Solaris and FEN myself, setting up CI, or any of that. What I'd like to see is yourself, but preferably multiple people, taking ownership of the FEN support.
The Solaris group would work with myself and hopefully some others on cross-platform API considerations (rename to, recursive watching, filtering, etc.). If there are bugs in the FEN support, I need to know there are people familiar with FEN that will triage issues and provide fixes. If/when we do a v2 API, I need to know there are people familiar with FEN that will help us move fsnotify forward on all platforms.
Honestly, I'd like to "recruit" small teams for each platform, but Solaris is a good place to start.
@isaacdavis In the near term I'd like to break fsnotify up into a few packages within the same repository. I've already begun some work in that direction with kqueue, by isolating the syscall wrappers.
This would allow someone to
Not saying this needs to happen before the PR is merged, but letting you know the direction I'm thinking of taking things.
The big question is how low-level is very low level. For example, the kqueue API takes file handles, not names of files to watch, so the kqueue wrapper may operate at that low level with very little in the way of amenities.
Part of my goal with this is to associate ownership to each subpackage. A small team managing each of fen, inotify, kqueue, fsevents -- possibly some people on multiple teams, and everyone participating in the common API design as well.
@nathany Thanks for your thorough responses. I'm happy to take ownership of FEN support and get involved in this project!
Regarding CI: After talking with some others at Joyent, we think the easiest solution would be to set up a Jenkins instance on a VM in the Joyent Public Cloud, and run the build and tests there. This would integrate well with Github. I would be your point of contact for administering this. Does that sound reasonable to you?
One other note: I only have the resources to provide illumos support - the build and tests would run on illumos, not Oracle Solaris. illumos diverged from Oracle Solaris in 2010, and there is no guarantee that the two operating systems share a common FEN API.
If this all sounds reasonable, let me know and I'll go ahead with addressing the code review and setting up CI. Thanks!
I think we'll document in the README who the teams are for each platform with respective GitHub handles, and perhaps figure out how to do some
Hi @nathany - In addition to addressing your comments, I've added a Jenkinsfile. I have a Jenkins server running at fsnotify-jenkins.englab.joyent.us that is configured to work with this Jenkinsfile - I've tested it with my fork of the repo. In order to integrate Jenkins with Github, the Jenkins server will need to authenticate as a user with write access to this repo. I can set you up a Jenkins account (which should be done anyway) and we can add your credentials, or, alternatively, if I'm given write access to this repo, I can set up auth using my own account. Let me know how best to move forward.
EDIT 2019-01-07: A third option would be to create a Github user (named e.g. "fsnotify automation") for the purpose of automating Jenkins integration. This makes more sense than explicitly linking Jenkins to someone's personal account.