Make package easier to package for Debian?

[petterreinholdtsen]

Hi. I suggested to add coz-profiler to Debian, and wonder if you might be interested in making this easier?

First and foremost, the package must be buildable without Internet access. The build process today clones a git repo and download some code using wget. Neither is going to work for Debian. Could you perhaps rewrite the build process to avoid such downloading?

Second, it would be easier to figure out which version number to use for the package if you versioned releases, either by tagging the git repo or by making tarball releases with version numbers. Would you be interested in doing this?

The request to add the package in Debian can be found using http://bugs.debian.org/830708 .

Subtasks:

• Move gh-pages contents into master as-is, and add build logic to build script.
• Include non-minimized versions of all javascript libraries next to the minimised versions.

Long term:

• Check on status of TypeScript 2.0.
  • If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
• Get d3-tip into Debian
• Get science.js into Debian

[ccurtsinger]

Thanks for submitting this. What is the preferred method for fetching dependencies? Would git submodules work? I would still like the make system to clone submodules if they aren't already checked out, but the Debian build could bypass this by running git clone --recursive.

[petterreinholdtsen]

[Charlie Curtsinger]

Thanks for submitting this. What is the preferred method for fetching
dependencies? Would git submodules work? I would still like the make
system to clone submodules if they aren't already checked out, but the
Debian build could bypass this by running git clone --recursive.

The preferred method is to fetch them from a mirror with Debian
packages. In other words, if you bring a copy of the Debian archive to
a isolated island in the ocean, you should still be able to build the
source packages. The autobuilders in Debian commonly run without
Internet access to verify that the source is complete and buildable
without connecting outside machines. So neither git clone nor git
submodules are going to work if the package should be included in
Debian. The build should use the source provided in the system by
Debian packages and the package source code.

For security reasons, it is also a goal of the Debian project to be able
to reproduce the build, ie be able to recreate the binary packages from
the sources. This is impossible if something besides the sources are
used to generate the binary packages.

Happy hacking
Petter Reinholdtsen

[petterreinholdtsen]

A Debian package probably would find it useful to have the source for the analyzing web site refered in issue #55 too.

[petterreinholdtsen]

Hi again. I was planning to look into getting coz into Debian, but I have not been able to get it working properly yet, and the build process is quite far away from what is needed in Debian. This make me unsure if trying to getting it into Debian is a good idea. What do you think?

[ccurtsinger]

As you've probably noticed, I haven't had a chance to address any of the major issues you ran into. I'm just now getting to work on coz this summer and plan to make quite a few improvements over the next couple months.

One important step in getting to a debian package is to get a libelfin debian package built as well. I need the libraries built with -fPIC so they can be used in a shared library, so this change would have to go upstream into a libelfin package. I'll submit the necessary pull requests (#58).

I propose we revisit packaging for debian in about a month, after I've had a chance to deal with the backlog of issues.

[petterreinholdtsen]

Btw, one question that will come up if the source is uploaded to debian is the copyright status of the Coz-Curtsinger-Berger-SOSP2015.pdf file. Is it licensed according to the BSD license as listed in LICENSE.md content indicates?

[ccurtsinger]

No, it actually shouldn't be in the repository. We (the authors) have retained copyright, but I believe in the publishing agreement we were required to grant exclusive rights to the ACM. This is inconsistent with requirements of the National Science Foundation for open access to federally-funded work, so it's unclear who has a right to distributed the article. I'm assuming everyone would prefer not to wander into that licensing wormhole.

[emeryberger]

Technically, here's the verbiage.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

However, ACM specifically allows us to post it under certain circumstances:

[...] post the Accepted Version of the Work on (1) the Author's home page, (2) the Owner's institutional repository, (3) any repository legally mandated by an agency funding the research on which the Work is based, and (4) any non-commercial repository or aggregation that does not duplicate ACM tables of contents, i.e., whose patterns of links do not substantially duplicate an ACM-copyrighted volume or issue. Non-commercial repositories are here understood as repositories owned by non-profit organizations that do not charge a fee for accessing deposited articles and that do not sell advertising or otherwise profit from serving articles.

So the right thing to do is to post a version on arXiv (or one of our home pages) and link to it (or request permission from ACM to include it in the repo).

[emeryberger]

I have gone ahead and requested permission though I doubt they will grant it. In the meanwhile, I will post it on arXiv.

[emeryberger]

Posted to arxiv. Should be up on August 15.

[petterreinholdtsen]

[Emery Berger notifications@github.com writes:

I have gone ahead and requested permission though I doubt they will
grant it. In the meanwhile, I will post it on arXiv.

Note, for Debian to accept the file into its archive, the terms of use
need to follow the Debian Free Software Guidelines,
<URL: https://www.debian.org/social_contract#guidelines >.

Happy hacking
Petter Reinholdtsen

[emeryberger]

I have removed the PDF and instead posted the link to the arXiv (which may be freely downloaded by anyone).

[petterreinholdtsen]

Thank you very much.

While trying to build a Debian package with the library and viewer, I ran into these errors from the build checking software (lintian):

E: coz-profiler source: source-is-missing viewer/profiles/swaptions.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/sqlite.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/memcached.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/fluidanimate.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/ferret.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/dedup.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/blackscholes.coz.js
E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

Is the error correct? Are the sources missing from the repository?

[petterreinholdtsen]

So, a month have passed. Any progress on preparing the package for inclusion in Debian?

libelfin entered Debian today, so that dependency is finally taken care of. But the profile viewer have several issues that need to be adressed before it can go into Debian

[petterreinholdtsen]

Hi. Any news on this? We would like to get coz into Debian, but there are several unsolved challenges, as you can see. :)

[jvilk]

@petterreinholdtsen I just opened a pull request making some of the desired changes to the profiler viewer, plus more enhancements.

The viewer/profiles sources are in the gh-pages branch. I've actually removed them in my pull request, since the new code fetches the .coz profiles directly instead.

E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

science.v1.min.js is a JavaScript library that the profiler viewer depends on. It's also in the gh-pages branch. My pull request also adds the other dependencies that we previously grabbed from a CDN (font-awesome, bootstrap, jQuery, and d3).

[petterreinholdtsen]

Hi. Any chance to include the source version of the javascript libraries in the source directory, and generate the packed version during build?

Also, any hope to provide a branch tag (aka version number) which include both the library and the viewer code? It would involve merging the gh-pages branch content into the master branch and tag the master branch, I believe.

Today I uploaded a draft debian package to Debian for review by the Debian ftpmasters. I used the script available from https://anonscm.debian.org/cgit/pkg-coz/coz-profiler.git/tree/debian/get-orig-source to make a release tarball, joining the master and gh-pages branches to get something I could use in Debian. I doubt it will be accepted because the source of the javascript libraries are missing, but wanted to give it a go to get feedback from the ftpmasters.

It might take a while before they have time to look at the package (weeks and months are normal), and I can upload new versions while we wait, so I would be very pleased if you could make coz easier to package in Debian while we wait for the review. :)

[petterreinholdtsen]

A list of relevant dates was just posted on https://lists.debian.org/debian-devel-announce/2016/11/msg00002.html . We are running short on time to get coz into the next Debian release. Do you plan to place the viewer as part of the source code and provide source there for the javascript libraries used by the viewer?

[emeryberger]

So to clarify, does this mean the deadline is in January?

[petterreinholdtsen]

[John Vilk]

@petterreinholdtsen I just opened a pull request making some of the
desired changes to the profiler viewer, plus more enhancements.

Very good to know there is progress. :)

The viewer/profiles sources are in the gh-pages branch. I've
actually removed them in my pull request, since the new code fetches
the .coz profiles directly instead.

I know it is in the gh-pages branch. I argue that it would be easier to
know which version of coz we are packaging for Debian if that viewer
code was in the master branch, and tagged with a common tag for both the
library and viewer. Then I could download a tarball for the tag from
github and use it to create the Debian package.

E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

science.v1.min.js is a JavaScript library that the profiler viewer
depends on. It's also in the gh-pages branch. My pull request also
adds the other dependencies that we previously grabbed from a CDN
(font-awesome, bootstrap, jQuery, and d3).

I suspect we talk past each other. The issue at hand is that the
javascript file in question is the equivalent of a binary, not source
code. Source code is in general in Debian considered the file with the
preferred form for modification. In other words, the javascript file
before minimisation (or browserified) should be included, and
preferabliy the build process should generate the browserified version.

The issue is discussed in <URL: https://bugs.debian.org/839570 >,
<URL: https://bugs.debian.org/840295 > and briefly in
<URL: https://wiki.debian.org/Javascript/Policy >.

The general idea is that someone stranded on a deserted island with a
DVD with the source code for Debian should have everything they need to
be able to fix bugs or modify the behaviour of code in Debian. If the
source for a javascript library is not included in the source code for
Debian, this become hard or impossible.

[Emery Berger]

So to clarify, does this mean the deadline is in January?

The upload deadline is most likely in early December. It leave some
days to fix last minute issues that are bound to show up when the code
is built for the first time on 20 architectures.

Happy hacking
Petter Reinholdtsen

[jvilk]

In other words, the javascript file before minimisation (or browserified) should be included, and preferabliy the build process should generate the browserified version.

If the original source is multiple modules that become bundled together before minification, is it sufficient to include the bundled-but-not-minified file? Or do you want us to preserve the original build process?

It looks like the following dependencies are already in Debian:

• d3
• jQuery
• font-awesome
• bootstrap

I have never packaged something for Debian, so I am not sure how, exactly, we can reference these. Do we just augment the build process to symlink these in from a known location when building for Debian? Is it acceptable for our build script to use separate logic for Debian vs. non-Debian, where these files would not be available?

The following are not present in Debian:

• d3-tip
  • Uses a single source file with no build step (other than minification), so it is easy to include in Coz.
• science.js
  • Uses a Makefile that calls Uglify on a set of source files.

Let me know, @petterreinholdtsen.

[jvilk]

Also, @petterreinholdtsen, our profiler viewer is compiled from TypeScript 2.0 to JavaScript. It looks like Debian's version of TypeScript is at 1.8. Backporting the source code to 1.8 would require adding a number of extra type annotations to the code, as we rely upon the tagged union feature of 2.0.

However, the JavaScript produced from TypeScript is not minified, easily readable, and just as modifiable as any other JavaScript file. Is that sufficient for inclusion in the package, or do we need to compile from the TypeScript source?

[petterreinholdtsen]

[John Vilk]

If the original source is multiple modules that become bundled
together before minification, is it sufficient to include the
bundled-but-not-minified file? Or do you want us to preserve the
original build process?

The preferred way is to only include the non-minimized javascript and
minimize it during build and before installation. If that is not
possible, my understanding is that a one time exception for Stretch is
given that the minimized version can be included in the source as long
as the non-minimized version is also included.

I have never packaged something for Debian, so I am not sure how,
exactly, we can reference these. Do we just augment the build process
to symlink these in from a known location when building for Debian? Is
it acceptable for our build script to use separate logic for Debian
vs. non-Debian, where these files would not be available?

<URL: https://anonscm.debian.org/cgit/pkg-coz/coz-profiler.git/tree/debian/rules >
show how I did it in the build process for the Debian package. I remove
the javascript modules that already exist in debian and insert a symlink
instead.

The following are not present in Debian:

• d3-tip
  • Uses a single source file with no build step (other than minification), so it is easy to include in Coz.
• science.js
  • Uses a Makefile that calls Uglify on a set of source files.

We should try to get them included in Debian.

Also, @petterreinholdtsen, our profiler viewer is compiled from
TypeScript 2.0 to JavaScript. It looks like Debian's version of
TypeScript is at
1.8. Backporting
the source code to 1.8 would require adding a number of extra type
annotations to the code, as we rely upon the tagged union feature of
2.0.

I noticed 2.0 is in experimental, and asked the maintainer in
<URL: https://bugs.debian.org/843804 > for more information.

But I must admit, javascript packaging is not my strong side. :)

Happy hacking
Petter Reinholdtsen

[jvilk]

Is this a reasonable checklist, then?

• Check on status of TypeScript 2.0.
  • If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
• Get d3-tip into Debian
• Get science.js into Debian
• Move gh-pages contents into master as-is, and add build logic to build script.
• For packaging for Debian, replace the JS libraries we depend on with symlinks.

Are you able to start the process for getting d3-tip and science.js into Debian? I am maxed out with my other duties right now to take responsibilities for those things, but I can work to get the profiler source into the master branch.

[petterreinholdtsen]

[John Vilk]

Is this a reasonable checklist, then?

I would split it in a short term and a long term list. Short Term:

• Move gh-pages contents into master as-is, and add build logic to build script.
• Include non-minimized versions of all javascript libraries next to the minimised versions.

Long term:

• Check on status of TypeScript 2.0.
  • If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
• Get d3-tip into Debian
• Get science.js into Debian

We will handle the debian packaging, so you do not need to think about
the symlinks.

Are you able to start the process for getting d3-tip and
science.js into Debian? I am maxed out with my other duties right
now to take responsibilities for those things, but I can work to get
the profiler source into the master branch.

I'll see what I can do. I would like to find someone in Debian with
more javascript packaging skills and get their opinions first, and get
their feedback.

Happy hacking
Petter Reinholdtsen

[petterreinholdtsen]

I've now asked the typescript maintainer for his plans for 2.0 in unstable, and registered a request to get d3-tip and science.js into Debian. See http://bugs.debian.org/830708 for the links.

[petterreinholdtsen]

I'm happy to report that the coz-profiler package was accepted into Debian yesterday. The package status can be seen on https://tracker.debian.org/pkg/coz-profiler .

[emeryberger]

So there are complaints about lack of source because we are using minimized JavaScript. Since most uses won't be over a network, it seems reasonable to think we could just use the un-minimized versions. @ccurtsinger ?

[petterreinholdtsen]

[Emery Berger]

So there are complaints about lack of source because we are using
minimized JavaScript. Since most uses won't be over a network, it
seems reasonable to think we could just use the un-minimized
versions. @ccurtsinger ?

This would solve the issue, definitely. What about providing the
un-minimized versions in the master branch, and the minimized versions
in the gh-pages branch?

Happy hacking
Petter Reinholdtsen

[llvilanova]

Regarding packaging, is there any reason to maintain ccutils as a separate repository? This forces debian to either manually copy its contents into debian's coz repository (breaking upstream's -- you -- history tracking due to additional fiddling), or to package ccutils as a separate package (which is a bit wasteful given its use).

[petterreinholdtsen]

Very good to see progress on this. The changes pulled in the last day move us a long way forward. Now if the uncompressed javascript libraries were included and release tags were used, we should be all set.

We are currently waiting for the broken binaries (on big endian architectures) to be removed from the Debian archive. Once this is done by the ftpmasters, the coz-profiler package will make it into 'testing' and become part of the next release (if no new RC issues pop up). I'm very happy libelfin and coz now include a simple self test, as it exposed the problems on big endian platforms. Without it, we would have shipped broken binaries in the next Debian release.

The status of libelfin and coz-profiler can be seen on https://qa.debian.org/developer.php?login=vilanova@ac.upc.edu .

[petterreinholdtsen]

The latest news is that the libelfin developer ported the library to big endian machines, and the library is now available on all release architectures in Debian. Thus we do no longer need to wait for the ftpmasters to remove broken binaries. I expect coz-profiler to make it into Debian Stretch (the future stable release) in 4 days.

Would you be willing to tag releases of coz? I notice there are currently no tags in the git repo.

Are there any news on including the source for the javascript libraries in the repo? I'm in contact with a friend who have the two remaining dependencies (d3-tip and science.js) as debian packages. We plan to upload them to Debian to be able to use debian dependencies for them.

[ccurtsinger]

Which version of D3 will be available? Version 4 introduced breaking API changes.

[petterreinholdtsen]

[Charlie Curtsinger]

Which version of D3 will be available? Version 4 introduced breaking API changes.

Version 3.5 so far, see <URL: https://tracker.debian.org/pkg/d3 >. -- Happy hacking Petter Reinholdtsen

[ccurtsinger]

Okay, I just pushed a version that has the uncompressed javascript and css sources. If everything looks good on your end, I can tag a release this evening.

[petterreinholdtsen]

[Charlie Curtsinger]

Okay, I just pushed a version that has the uncompressed javascript and css sources. If everything looks good on your end, I can tag a release this evening.

Both 'make' and 'make check' work for me with the latest version, so it look good to me. But I commented out the npm call in viewer/Makefile which do not seem to do anything I need to get the viewer working. What exactly is npm install doing?

…

-- Happy hacking Petter Reinholdtsen

[ccurtsinger]

The npm install command will go grab dependencies for the viewer. I'm not terribly comfortable working with npm, so my inclination is to switch to a makefile-based typescript compile instead. That should fit better with the Debian model, where there should be no network dependencies during the build.

[llvilanova]

Hi Peter, can any of the tasks of this bug be marked as done?

[petterreinholdtsen]

I can't really speak for Peter, but I had a look and d3-tip and science both seem to be in Debian now. I hope the analysis web pages can be included in the official release tarball, but have not seen any progress there. As for TypeScript, I have no idea what the status is.

[llvilanova]

The first point in this task list should be marked as done (the gh-pages branch was moved into the viewer directory).

The second one seems it can be marked as done too; all libraries in the viewer dir seem to have a non-minimised version.

[emeryberger]

Marked those tasks as done.

[ccurtsinger]

Coz is (and has been) in Debian, so this must have been resolved. Future changes can be filed as separate issues.