New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make package easier to package for Debian? #48

Open
petterreinholdtsen opened this Issue Jul 10, 2016 · 41 comments

Comments

Projects
None yet
5 participants
@petterreinholdtsen
Contributor

petterreinholdtsen commented Jul 10, 2016

Hi. I suggested to add coz-profiler to Debian, and wonder if you might be interested in making this easier?

First and foremost, the package must be buildable without Internet access. The build process today clones a git repo and download some code using wget. Neither is going to work for Debian. Could you perhaps rewrite the build process to avoid such downloading?

Second, it would be easier to figure out which version number to use for the package if you versioned releases, either by tagging the git repo or by making tarball releases with version numbers. Would you be interested in doing this?

The request to add the package in Debian can be found using http://bugs.debian.org/830708 .

Subtasks:

  • Move gh-pages contents into master as-is, and add build logic to build script.
  • Include non-minimized versions of all javascript libraries next to the minimised versions.

Long term:

  • Check on status of TypeScript 2.0.
    • If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
  • Get d3-tip into Debian
  • Get science.js into Debian
@ccurtsinger

This comment has been minimized.

Show comment
Hide comment
@ccurtsinger

ccurtsinger Jul 14, 2016

Member

Thanks for submitting this. What is the preferred method for fetching dependencies? Would git submodules work? I would still like the make system to clone submodules if they aren't already checked out, but the Debian build could bypass this by running git clone --recursive.

Member

ccurtsinger commented Jul 14, 2016

Thanks for submitting this. What is the preferred method for fetching dependencies? Would git submodules work? I would still like the make system to clone submodules if they aren't already checked out, but the Debian build could bypass this by running git clone --recursive.

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Jul 14, 2016

Contributor

[Charlie Curtsinger]

Thanks for submitting this. What is the preferred method for fetching
dependencies? Would git submodules work? I would still like the make
system to clone submodules if they aren't already checked out, but the
Debian build could bypass this by running git clone --recursive.

The preferred method is to fetch them from a mirror with Debian
packages. In other words, if you bring a copy of the Debian archive to
a isolated island in the ocean, you should still be able to build the
source packages. The autobuilders in Debian commonly run without
Internet access to verify that the source is complete and buildable
without connecting outside machines. So neither git clone nor git
submodules are going to work if the package should be included in
Debian. The build should use the source provided in the system by
Debian packages and the package source code.

For security reasons, it is also a goal of the Debian project to be able
to reproduce the build, ie be able to recreate the binary packages from
the sources. This is impossible if something besides the sources are
used to generate the binary packages.

Happy hacking
Petter Reinholdtsen

Contributor

petterreinholdtsen commented Jul 14, 2016

[Charlie Curtsinger]

Thanks for submitting this. What is the preferred method for fetching
dependencies? Would git submodules work? I would still like the make
system to clone submodules if they aren't already checked out, but the
Debian build could bypass this by running git clone --recursive.

The preferred method is to fetch them from a mirror with Debian
packages. In other words, if you bring a copy of the Debian archive to
a isolated island in the ocean, you should still be able to build the
source packages. The autobuilders in Debian commonly run without
Internet access to verify that the source is complete and buildable
without connecting outside machines. So neither git clone nor git
submodules are going to work if the package should be included in
Debian. The build should use the source provided in the system by
Debian packages and the package source code.

For security reasons, it is also a goal of the Debian project to be able
to reproduce the build, ie be able to recreate the binary packages from
the sources. This is impossible if something besides the sources are
used to generate the binary packages.

Happy hacking
Petter Reinholdtsen

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Jul 18, 2016

Contributor

A Debian package probably would find it useful to have the source for the analyzing web site refered in issue #55 too.

Contributor

petterreinholdtsen commented Jul 18, 2016

A Debian package probably would find it useful to have the source for the analyzing web site refered in issue #55 too.

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Aug 8, 2016

Contributor

Hi again. I was planning to look into getting coz into Debian, but I have not been able to get it working properly yet, and the build process is quite far away from what is needed in Debian. This make me unsure if trying to getting it into Debian is a good idea. What do you think?

Contributor

petterreinholdtsen commented Aug 8, 2016

Hi again. I was planning to look into getting coz into Debian, but I have not been able to get it working properly yet, and the build process is quite far away from what is needed in Debian. This make me unsure if trying to getting it into Debian is a good idea. What do you think?

@ccurtsinger ccurtsinger added the on hold label Aug 10, 2016

@ccurtsinger

This comment has been minimized.

Show comment
Hide comment
@ccurtsinger

ccurtsinger Aug 10, 2016

Member

As you've probably noticed, I haven't had a chance to address any of the major issues you ran into. I'm just now getting to work on coz this summer and plan to make quite a few improvements over the next couple months.

One important step in getting to a debian package is to get a libelfin debian package built as well. I need the libraries built with -fPIC so they can be used in a shared library, so this change would have to go upstream into a libelfin package. I'll submit the necessary pull requests (#58).

I propose we revisit packaging for debian in about a month, after I've had a chance to deal with the backlog of issues.

Member

ccurtsinger commented Aug 10, 2016

As you've probably noticed, I haven't had a chance to address any of the major issues you ran into. I'm just now getting to work on coz this summer and plan to make quite a few improvements over the next couple months.

One important step in getting to a debian package is to get a libelfin debian package built as well. I need the libraries built with -fPIC so they can be used in a shared library, so this change would have to go upstream into a libelfin package. I'll submit the necessary pull requests (#58).

I propose we revisit packaging for debian in about a month, after I've had a chance to deal with the backlog of issues.

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Aug 10, 2016

Contributor

Btw, one question that will come up if the source is uploaded to debian is the copyright status of the Coz-Curtsinger-Berger-SOSP2015.pdf file. Is it licensed according to the BSD license as listed in LICENSE.md content indicates?

Contributor

petterreinholdtsen commented Aug 10, 2016

Btw, one question that will come up if the source is uploaded to debian is the copyright status of the Coz-Curtsinger-Berger-SOSP2015.pdf file. Is it licensed according to the BSD license as listed in LICENSE.md content indicates?

@ccurtsinger

This comment has been minimized.

Show comment
Hide comment
@ccurtsinger

ccurtsinger Aug 10, 2016

Member

No, it actually shouldn't be in the repository. We (the authors) have retained copyright, but I believe in the publishing agreement we were required to grant exclusive rights to the ACM. This is inconsistent with requirements of the National Science Foundation for open access to federally-funded work, so it's unclear who has a right to distributed the article. I'm assuming everyone would prefer not to wander into that licensing wormhole.

Member

ccurtsinger commented Aug 10, 2016

No, it actually shouldn't be in the repository. We (the authors) have retained copyright, but I believe in the publishing agreement we were required to grant exclusive rights to the ACM. This is inconsistent with requirements of the National Science Foundation for open access to federally-funded work, so it's unclear who has a right to distributed the article. I'm assuming everyone would prefer not to wander into that licensing wormhole.

@emeryberger

This comment has been minimized.

Show comment
Hide comment
@emeryberger

emeryberger Aug 12, 2016

Member

Technically, here's the verbiage.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

However, ACM specifically allows us to post it under certain circumstances:

[...] post the Accepted Version of the Work on (1) the Author's home page, (2) the Owner's institutional repository, (3) any repository legally mandated by an agency funding the research on which the Work is based, and (4) any non-commercial repository or aggregation that does not duplicate ACM tables of contents, i.e., whose patterns of links do not substantially duplicate an ACM-copyrighted volume or issue. Non-commercial repositories are here understood as repositories owned by non-profit organizations that do not charge a fee for accessing deposited articles and that do not sell advertising or otherwise profit from serving articles.

So the right thing to do is to post a version on arXiv (or one of our home pages) and link to it (or request permission from ACM to include it in the repo).

Member

emeryberger commented Aug 12, 2016

Technically, here's the verbiage.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

However, ACM specifically allows us to post it under certain circumstances:

[...] post the Accepted Version of the Work on (1) the Author's home page, (2) the Owner's institutional repository, (3) any repository legally mandated by an agency funding the research on which the Work is based, and (4) any non-commercial repository or aggregation that does not duplicate ACM tables of contents, i.e., whose patterns of links do not substantially duplicate an ACM-copyrighted volume or issue. Non-commercial repositories are here understood as repositories owned by non-profit organizations that do not charge a fee for accessing deposited articles and that do not sell advertising or otherwise profit from serving articles.

So the right thing to do is to post a version on arXiv (or one of our home pages) and link to it (or request permission from ACM to include it in the repo).

@emeryberger

This comment has been minimized.

Show comment
Hide comment
@emeryberger

emeryberger Aug 12, 2016

Member

I have gone ahead and requested permission though I doubt they will grant it. In the meanwhile, I will post it on arXiv.

Member

emeryberger commented Aug 12, 2016

I have gone ahead and requested permission though I doubt they will grant it. In the meanwhile, I will post it on arXiv.

@emeryberger

This comment has been minimized.

Show comment
Hide comment
@emeryberger

emeryberger Aug 12, 2016

Member

Posted to arxiv. Should be up on August 15.

Member

emeryberger commented Aug 12, 2016

Posted to arxiv. Should be up on August 15.

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Aug 12, 2016

Contributor

[Emery Berger notifications@github.com writes:

I have gone ahead and requested permission though I doubt they will
grant it. In the meanwhile, I will post it on arXiv.

Note, for Debian to accept the file into its archive, the terms of use
need to follow the Debian Free Software Guidelines,
<URL: https://www.debian.org/social_contract#guidelines >.

Happy hacking
Petter Reinholdtsen

Contributor

petterreinholdtsen commented Aug 12, 2016

[Emery Berger notifications@github.com writes:

I have gone ahead and requested permission though I doubt they will
grant it. In the meanwhile, I will post it on arXiv.

Note, for Debian to accept the file into its archive, the terms of use
need to follow the Debian Free Software Guidelines,
<URL: https://www.debian.org/social_contract#guidelines >.

Happy hacking
Petter Reinholdtsen

@emeryberger

This comment has been minimized.

Show comment
Hide comment
@emeryberger

emeryberger Aug 15, 2016

Member

I have removed the PDF and instead posted the link to the arXiv (which may be freely downloaded by anyone).

Member

emeryberger commented Aug 15, 2016

I have removed the PDF and instead posted the link to the arXiv (which may be freely downloaded by anyone).

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Aug 15, 2016

Contributor

Thank you very much.

While trying to build a Debian package with the library and viewer, I ran into these errors from the build checking software (lintian):

E: coz-profiler source: source-is-missing viewer/profiles/swaptions.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/sqlite.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/memcached.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/fluidanimate.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/ferret.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/dedup.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/blackscholes.coz.js
E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

Is the error correct? Are the sources missing from the repository?

Contributor

petterreinholdtsen commented Aug 15, 2016

Thank you very much.

While trying to build a Debian package with the library and viewer, I ran into these errors from the build checking software (lintian):

E: coz-profiler source: source-is-missing viewer/profiles/swaptions.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/sqlite.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/memcached.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/fluidanimate.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/ferret.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/dedup.coz.js
E: coz-profiler source: source-is-missing viewer/profiles/blackscholes.coz.js
E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

Is the error correct? Are the sources missing from the repository?

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Sep 13, 2016

Contributor

So, a month have passed. Any progress on preparing the package for inclusion in Debian?

libelfin entered Debian today, so that dependency is finally taken care of. But the profile viewer have several issues that need to be adressed before it can go into Debian

Contributor

petterreinholdtsen commented Sep 13, 2016

So, a month have passed. Any progress on preparing the package for inclusion in Debian?

libelfin entered Debian today, so that dependency is finally taken care of. But the profile viewer have several issues that need to be adressed before it can go into Debian

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Oct 11, 2016

Contributor

Hi. Any news on this? We would like to get coz into Debian, but there are several unsolved challenges, as you can see. :)

Contributor

petterreinholdtsen commented Oct 11, 2016

Hi. Any news on this? We would like to get coz into Debian, but there are several unsolved challenges, as you can see. :)

@jvilk

This comment has been minimized.

Show comment
Hide comment
@jvilk

jvilk Oct 11, 2016

Member

@petterreinholdtsen I just opened a pull request making some of the desired changes to the profiler viewer, plus more enhancements.

The viewer/profiles sources are in the gh-pages branch. I've actually removed them in my pull request, since the new code fetches the .coz profiles directly instead.

E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

science.v1.min.js is a JavaScript library that the profiler viewer depends on. It's also in the gh-pages branch. My pull request also adds the other dependencies that we previously grabbed from a CDN (font-awesome, bootstrap, jQuery, and d3).

Member

jvilk commented Oct 11, 2016

@petterreinholdtsen I just opened a pull request making some of the desired changes to the profiler viewer, plus more enhancements.

The viewer/profiles sources are in the gh-pages branch. I've actually removed them in my pull request, since the new code fetches the .coz profiles directly instead.

E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

science.v1.min.js is a JavaScript library that the profiler viewer depends on. It's also in the gh-pages branch. My pull request also adds the other dependencies that we previously grabbed from a CDN (font-awesome, bootstrap, jQuery, and d3).

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Oct 31, 2016

Contributor

Hi. Any chance to include the source version of the javascript libraries in the source directory, and generate the packed version during build?

Also, any hope to provide a branch tag (aka version number) which include both the library and the viewer code? It would involve merging the gh-pages branch content into the master branch and tag the master branch, I believe.

Today I uploaded a draft debian package to Debian for review by the Debian ftpmasters. I used the script available from https://anonscm.debian.org/cgit/pkg-coz/coz-profiler.git/tree/debian/get-orig-source to make a release tarball, joining the master and gh-pages branches to get something I could use in Debian. I doubt it will be accepted because the source of the javascript libraries are missing, but wanted to give it a go to get feedback from the ftpmasters.

It might take a while before they have time to look at the package (weeks and months are normal), and I can upload new versions while we wait, so I would be very pleased if you could make coz easier to package in Debian while we wait for the review. :)

Contributor

petterreinholdtsen commented Oct 31, 2016

Hi. Any chance to include the source version of the javascript libraries in the source directory, and generate the packed version during build?

Also, any hope to provide a branch tag (aka version number) which include both the library and the viewer code? It would involve merging the gh-pages branch content into the master branch and tag the master branch, I believe.

Today I uploaded a draft debian package to Debian for review by the Debian ftpmasters. I used the script available from https://anonscm.debian.org/cgit/pkg-coz/coz-profiler.git/tree/debian/get-orig-source to make a release tarball, joining the master and gh-pages branches to get something I could use in Debian. I doubt it will be accepted because the source of the javascript libraries are missing, but wanted to give it a go to get feedback from the ftpmasters.

It might take a while before they have time to look at the package (weeks and months are normal), and I can upload new versions while we wait, so I would be very pleased if you could make coz easier to package in Debian while we wait for the review. :)

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Nov 5, 2016

Contributor

A list of relevant dates was just posted on https://lists.debian.org/debian-devel-announce/2016/11/msg00002.html . We are running short on time to get coz into the next Debian release. Do you plan to place the viewer as part of the source code and provide source there for the javascript libraries used by the viewer?

Contributor

petterreinholdtsen commented Nov 5, 2016

A list of relevant dates was just posted on https://lists.debian.org/debian-devel-announce/2016/11/msg00002.html . We are running short on time to get coz into the next Debian release. Do you plan to place the viewer as part of the source code and provide source there for the javascript libraries used by the viewer?

@emeryberger

This comment has been minimized.

Show comment
Hide comment
@emeryberger

emeryberger Nov 8, 2016

Member

So to clarify, does this mean the deadline is in January?

Member

emeryberger commented Nov 8, 2016

So to clarify, does this mean the deadline is in January?

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Nov 8, 2016

Contributor

[John Vilk]

@petterreinholdtsen I just opened a pull request making some of the
desired changes to the profiler viewer, plus more enhancements.

Very good to know there is progress. :)

The viewer/profiles sources are in the gh-pages branch. I've
actually removed them in my pull request, since the new code fetches
the .coz profiles directly instead.

I know it is in the gh-pages branch. I argue that it would be easier to
know which version of coz we are packaging for Debian if that viewer
code was in the master branch, and tagged with a common tag for both the
library and viewer. Then I could download a tarball for the tag from
github and use it to create the Debian package.

E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

science.v1.min.js is a JavaScript library that the profiler viewer
depends on. It's also in the gh-pages branch. My pull request also
adds the other dependencies that we previously grabbed from a CDN
(font-awesome, bootstrap, jQuery, and d3).

I suspect we talk past each other. The issue at hand is that the
javascript file in question is the equivalent of a binary, not source
code. Source code is in general in Debian considered the file with the
preferred form for modification. In other words, the javascript file
before minimisation (or browserified) should be included, and
preferabliy the build process should generate the browserified version.

The issue is discussed in <URL: https://bugs.debian.org/839570 >,
<URL: https://bugs.debian.org/840295 > and briefly in
<URL: https://wiki.debian.org/Javascript/Policy >.

The general idea is that someone stranded on a deserted island with a
DVD with the source code for Debian should have everything they need to
be able to fix bugs or modify the behaviour of code in Debian. If the
source for a javascript library is not included in the source code for
Debian, this become hard or impossible.

[Emery Berger]

So to clarify, does this mean the deadline is in January?

The upload deadline is most likely in early December. It leave some
days to fix last minute issues that are bound to show up when the code
is built for the first time on 20 architectures.

Happy hacking
Petter Reinholdtsen

Contributor

petterreinholdtsen commented Nov 8, 2016

[John Vilk]

@petterreinholdtsen I just opened a pull request making some of the
desired changes to the profiler viewer, plus more enhancements.

Very good to know there is progress. :)

The viewer/profiles sources are in the gh-pages branch. I've
actually removed them in my pull request, since the new code fetches
the .coz profiles directly instead.

I know it is in the gh-pages branch. I argue that it would be easier to
know which version of coz we are packaging for Debian if that viewer
code was in the master branch, and tagged with a common tag for both the
library and viewer. Then I could download a tarball for the tag from
github and use it to create the Debian package.

E: coz-profiler source: source-is-missing viewer/lib/science/science.v1.min.js

science.v1.min.js is a JavaScript library that the profiler viewer
depends on. It's also in the gh-pages branch. My pull request also
adds the other dependencies that we previously grabbed from a CDN
(font-awesome, bootstrap, jQuery, and d3).

I suspect we talk past each other. The issue at hand is that the
javascript file in question is the equivalent of a binary, not source
code. Source code is in general in Debian considered the file with the
preferred form for modification. In other words, the javascript file
before minimisation (or browserified) should be included, and
preferabliy the build process should generate the browserified version.

The issue is discussed in <URL: https://bugs.debian.org/839570 >,
<URL: https://bugs.debian.org/840295 > and briefly in
<URL: https://wiki.debian.org/Javascript/Policy >.

The general idea is that someone stranded on a deserted island with a
DVD with the source code for Debian should have everything they need to
be able to fix bugs or modify the behaviour of code in Debian. If the
source for a javascript library is not included in the source code for
Debian, this become hard or impossible.

[Emery Berger]

So to clarify, does this mean the deadline is in January?

The upload deadline is most likely in early December. It leave some
days to fix last minute issues that are bound to show up when the code
is built for the first time on 20 architectures.

Happy hacking
Petter Reinholdtsen

@jvilk

This comment has been minimized.

Show comment
Hide comment
@jvilk

jvilk Nov 9, 2016

Member

In other words, the javascript file before minimisation (or browserified) should be included, and preferabliy the build process should generate the browserified version.

If the original source is multiple modules that become bundled together before minification, is it sufficient to include the bundled-but-not-minified file? Or do you want us to preserve the original build process?

It looks like the following dependencies are already in Debian:

I have never packaged something for Debian, so I am not sure how, exactly, we can reference these. Do we just augment the build process to symlink these in from a known location when building for Debian? Is it acceptable for our build script to use separate logic for Debian vs. non-Debian, where these files would not be available?

The following are not present in Debian:

  • d3-tip
    • Uses a single source file with no build step (other than minification), so it is easy to include in Coz.
  • science.js
    • Uses a Makefile that calls Uglify on a set of source files.

Let me know, @petterreinholdtsen.

Member

jvilk commented Nov 9, 2016

In other words, the javascript file before minimisation (or browserified) should be included, and preferabliy the build process should generate the browserified version.

If the original source is multiple modules that become bundled together before minification, is it sufficient to include the bundled-but-not-minified file? Or do you want us to preserve the original build process?

It looks like the following dependencies are already in Debian:

I have never packaged something for Debian, so I am not sure how, exactly, we can reference these. Do we just augment the build process to symlink these in from a known location when building for Debian? Is it acceptable for our build script to use separate logic for Debian vs. non-Debian, where these files would not be available?

The following are not present in Debian:

  • d3-tip
    • Uses a single source file with no build step (other than minification), so it is easy to include in Coz.
  • science.js
    • Uses a Makefile that calls Uglify on a set of source files.

Let me know, @petterreinholdtsen.

@jvilk

This comment has been minimized.

Show comment
Hide comment
@jvilk

jvilk Nov 9, 2016

Member

Also, @petterreinholdtsen, our profiler viewer is compiled from TypeScript 2.0 to JavaScript. It looks like Debian's version of TypeScript is at 1.8. Backporting the source code to 1.8 would require adding a number of extra type annotations to the code, as we rely upon the tagged union feature of 2.0.

However, the JavaScript produced from TypeScript is not minified, easily readable, and just as modifiable as any other JavaScript file. Is that sufficient for inclusion in the package, or do we need to compile from the TypeScript source?

Member

jvilk commented Nov 9, 2016

Also, @petterreinholdtsen, our profiler viewer is compiled from TypeScript 2.0 to JavaScript. It looks like Debian's version of TypeScript is at 1.8. Backporting the source code to 1.8 would require adding a number of extra type annotations to the code, as we rely upon the tagged union feature of 2.0.

However, the JavaScript produced from TypeScript is not minified, easily readable, and just as modifiable as any other JavaScript file. Is that sufficient for inclusion in the package, or do we need to compile from the TypeScript source?

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Nov 9, 2016

Contributor

[John Vilk]

If the original source is multiple modules that become bundled
together before minification, is it sufficient to include the
bundled-but-not-minified file? Or do you want us to preserve the
original build process?

The preferred way is to only include the non-minimized javascript and
minimize it during build and before installation. If that is not
possible, my understanding is that a one time exception for Stretch is
given that the minimized version can be included in the source as long
as the non-minimized version is also included.

I have never packaged something for Debian, so I am not sure how,
exactly, we can reference these. Do we just augment the build process
to symlink these in from a known location when building for Debian? Is
it acceptable for our build script to use separate logic for Debian
vs. non-Debian, where these files would not be available?

<URL: https://anonscm.debian.org/cgit/pkg-coz/coz-profiler.git/tree/debian/rules >
show how I did it in the build process for the Debian package. I remove
the javascript modules that already exist in debian and insert a symlink
instead.

The following are not present in Debian:

  • d3-tip
    • Uses a single source file with no build step (other than minification), so it is easy to include in Coz.
  • science.js
    • Uses a Makefile that calls Uglify on a set of source files.

We should try to get them included in Debian.

Also, @petterreinholdtsen, our profiler viewer is compiled from
TypeScript 2.0 to JavaScript. It looks like Debian's version of
TypeScript is at
1.8
. Backporting
the source code to 1.8 would require adding a number of extra type
annotations to the code, as we rely upon the tagged union feature of
2.0.

I noticed 2.0 is in experimental, and asked the maintainer in
<URL: https://bugs.debian.org/843804 > for more information.

But I must admit, javascript packaging is not my strong side. :)

Happy hacking
Petter Reinholdtsen

Contributor

petterreinholdtsen commented Nov 9, 2016

[John Vilk]

If the original source is multiple modules that become bundled
together before minification, is it sufficient to include the
bundled-but-not-minified file? Or do you want us to preserve the
original build process?

The preferred way is to only include the non-minimized javascript and
minimize it during build and before installation. If that is not
possible, my understanding is that a one time exception for Stretch is
given that the minimized version can be included in the source as long
as the non-minimized version is also included.

I have never packaged something for Debian, so I am not sure how,
exactly, we can reference these. Do we just augment the build process
to symlink these in from a known location when building for Debian? Is
it acceptable for our build script to use separate logic for Debian
vs. non-Debian, where these files would not be available?

<URL: https://anonscm.debian.org/cgit/pkg-coz/coz-profiler.git/tree/debian/rules >
show how I did it in the build process for the Debian package. I remove
the javascript modules that already exist in debian and insert a symlink
instead.

The following are not present in Debian:

  • d3-tip
    • Uses a single source file with no build step (other than minification), so it is easy to include in Coz.
  • science.js
    • Uses a Makefile that calls Uglify on a set of source files.

We should try to get them included in Debian.

Also, @petterreinholdtsen, our profiler viewer is compiled from
TypeScript 2.0 to JavaScript. It looks like Debian's version of
TypeScript is at
1.8
. Backporting
the source code to 1.8 would require adding a number of extra type
annotations to the code, as we rely upon the tagged union feature of
2.0.

I noticed 2.0 is in experimental, and asked the maintainer in
<URL: https://bugs.debian.org/843804 > for more information.

But I must admit, javascript packaging is not my strong side. :)

Happy hacking
Petter Reinholdtsen

@jvilk

This comment has been minimized.

Show comment
Hide comment
@jvilk

jvilk Nov 9, 2016

Member

Is this a reasonable checklist, then?

  • Check on status of TypeScript 2.0.
    • If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
  • Get d3-tip into Debian
  • Get science.js into Debian
  • Move gh-pages contents into master as-is, and add build logic to build script.
  • For packaging for Debian, replace the JS libraries we depend on with symlinks.

Are you able to start the process for getting d3-tip and science.js into Debian? I am maxed out with my other duties right now to take responsibilities for those things, but I can work to get the profiler source into the master branch.

Member

jvilk commented Nov 9, 2016

Is this a reasonable checklist, then?

  • Check on status of TypeScript 2.0.
    • If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
  • Get d3-tip into Debian
  • Get science.js into Debian
  • Move gh-pages contents into master as-is, and add build logic to build script.
  • For packaging for Debian, replace the JS libraries we depend on with symlinks.

Are you able to start the process for getting d3-tip and science.js into Debian? I am maxed out with my other duties right now to take responsibilities for those things, but I can work to get the profiler source into the master branch.

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Nov 9, 2016

Contributor

[John Vilk]

Is this a reasonable checklist, then?

I would split it in a short term and a long term list. Short Term:

  • Move gh-pages contents into master as-is, and add build logic to build script.
  • Include non-minimized versions of all javascript libraries next to the minimised versions.

Long term:

  • Check on status of TypeScript 2.0.
    • If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
  • Get d3-tip into Debian
  • Get science.js into Debian

We will handle the debian packaging, so you do not need to think about
the symlinks.

Are you able to start the process for getting d3-tip and
science.js into Debian? I am maxed out with my other duties right
now to take responsibilities for those things, but I can work to get
the profiler source into the master branch.

I'll see what I can do. I would like to find someone in Debian with
more javascript packaging skills and get their opinions first, and get
their feedback.

Happy hacking
Petter Reinholdtsen

Contributor

petterreinholdtsen commented Nov 9, 2016

[John Vilk]

Is this a reasonable checklist, then?

I would split it in a short term and a long term list. Short Term:

  • Move gh-pages contents into master as-is, and add build logic to build script.
  • Include non-minimized versions of all javascript libraries next to the minimised versions.

Long term:

  • Check on status of TypeScript 2.0.
    • If we cannot use it, we can just include the JS version of the profiler source. It's non-ideal, but still human-readable, commented, and modifiable.
  • Get d3-tip into Debian
  • Get science.js into Debian

We will handle the debian packaging, so you do not need to think about
the symlinks.

Are you able to start the process for getting d3-tip and
science.js into Debian? I am maxed out with my other duties right
now to take responsibilities for those things, but I can work to get
the profiler source into the master branch.

I'll see what I can do. I would like to find someone in Debian with
more javascript packaging skills and get their opinions first, and get
their feedback.

Happy hacking
Petter Reinholdtsen

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Nov 11, 2016

Contributor

I've now asked the typescript maintainer for his plans for 2.0 in unstable, and registered a request to get d3-tip and science.js into Debian. See http://bugs.debian.org/830708 for the links.

Contributor

petterreinholdtsen commented Nov 11, 2016

I've now asked the typescript maintainer for his plans for 2.0 in unstable, and registered a request to get d3-tip and science.js into Debian. See http://bugs.debian.org/830708 for the links.

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Nov 12, 2016

Contributor

I'm happy to report that the coz-profiler package was accepted into Debian yesterday. The package status can be seen on https://tracker.debian.org/pkg/coz-profiler .

Contributor

petterreinholdtsen commented Nov 12, 2016

I'm happy to report that the coz-profiler package was accepted into Debian yesterday. The package status can be seen on https://tracker.debian.org/pkg/coz-profiler .

@emeryberger

This comment has been minimized.

Show comment
Hide comment
@emeryberger

emeryberger Nov 12, 2016

Member

So there are complaints about lack of source because we are using minimized JavaScript. Since most uses won't be over a network, it seems reasonable to think we could just use the un-minimized versions. @ccurtsinger ?

Member

emeryberger commented Nov 12, 2016

So there are complaints about lack of source because we are using minimized JavaScript. Since most uses won't be over a network, it seems reasonable to think we could just use the un-minimized versions. @ccurtsinger ?

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Nov 16, 2016

Contributor

[Emery Berger]

So there are complaints about lack of source because we are using
minimized JavaScript. Since most uses won't be over a network, it
seems reasonable to think we could just use the un-minimized
versions. @ccurtsinger ?

This would solve the issue, definitely. What about providing the
un-minimized versions in the master branch, and the minimized versions
in the gh-pages branch?

Happy hacking
Petter Reinholdtsen

Contributor

petterreinholdtsen commented Nov 16, 2016

[Emery Berger]

So there are complaints about lack of source because we are using
minimized JavaScript. Since most uses won't be over a network, it
seems reasonable to think we could just use the un-minimized
versions. @ccurtsinger ?

This would solve the issue, definitely. What about providing the
un-minimized versions in the master branch, and the minimized versions
in the gh-pages branch?

Happy hacking
Petter Reinholdtsen

@llvilanova

This comment has been minimized.

Show comment
Hide comment
@llvilanova

llvilanova Nov 17, 2016

Contributor

Regarding packaging, is there any reason to maintain ccutils as a separate repository? This forces debian to either manually copy its contents into debian's coz repository (breaking upstream's -- you -- history tracking due to additional fiddling), or to package ccutils as a separate package (which is a bit wasteful given its use).

Contributor

llvilanova commented Nov 17, 2016

Regarding packaging, is there any reason to maintain ccutils as a separate repository? This forces debian to either manually copy its contents into debian's coz repository (breaking upstream's -- you -- history tracking due to additional fiddling), or to package ccutils as a separate package (which is a bit wasteful given its use).

ccurtsinger added a commit that referenced this issue Nov 23, 2016

@ccurtsinger ccurtsinger added in progress and removed on hold labels Nov 23, 2016

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Nov 23, 2016

Contributor

Very good to see progress on this. The changes pulled in the last day move us a long way forward. Now if the uncompressed javascript libraries were included and release tags were used, we should be all set.

We are currently waiting for the broken binaries (on big endian architectures) to be removed from the Debian archive. Once this is done by the ftpmasters, the coz-profiler package will make it into 'testing' and become part of the next release (if no new RC issues pop up). I'm very happy libelfin and coz now include a simple self test, as it exposed the problems on big endian platforms. Without it, we would have shipped broken binaries in the next Debian release.

The status of libelfin and coz-profiler can be seen on https://qa.debian.org/developer.php?login=vilanova@ac.upc.edu .

Contributor

petterreinholdtsen commented Nov 23, 2016

Very good to see progress on this. The changes pulled in the last day move us a long way forward. Now if the uncompressed javascript libraries were included and release tags were used, we should be all set.

We are currently waiting for the broken binaries (on big endian architectures) to be removed from the Debian archive. Once this is done by the ftpmasters, the coz-profiler package will make it into 'testing' and become part of the next release (if no new RC issues pop up). I'm very happy libelfin and coz now include a simple self test, as it exposed the problems on big endian platforms. Without it, we would have shipped broken binaries in the next Debian release.

The status of libelfin and coz-profiler can be seen on https://qa.debian.org/developer.php?login=vilanova@ac.upc.edu .

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Dec 5, 2016

Contributor

The latest news is that the libelfin developer ported the library to big endian machines, and the library is now available on all release architectures in Debian. Thus we do no longer need to wait for the ftpmasters to remove broken binaries. I expect coz-profiler to make it into Debian Stretch (the future stable release) in 4 days.

Would you be willing to tag releases of coz? I notice there are currently no tags in the git repo.

Are there any news on including the source for the javascript libraries in the repo? I'm in contact with a friend who have the two remaining dependencies (d3-tip and science.js) as debian packages. We plan to upload them to Debian to be able to use debian dependencies for them.

Contributor

petterreinholdtsen commented Dec 5, 2016

The latest news is that the libelfin developer ported the library to big endian machines, and the library is now available on all release architectures in Debian. Thus we do no longer need to wait for the ftpmasters to remove broken binaries. I expect coz-profiler to make it into Debian Stretch (the future stable release) in 4 days.

Would you be willing to tag releases of coz? I notice there are currently no tags in the git repo.

Are there any news on including the source for the javascript libraries in the repo? I'm in contact with a friend who have the two remaining dependencies (d3-tip and science.js) as debian packages. We plan to upload them to Debian to be able to use debian dependencies for them.

@ccurtsinger

This comment has been minimized.

Show comment
Hide comment
@ccurtsinger

ccurtsinger Dec 7, 2016

Member

Which version of D3 will be available? Version 4 introduced breaking API changes.

Member

ccurtsinger commented Dec 7, 2016

Which version of D3 will be available? Version 4 introduced breaking API changes.

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Dec 8, 2016

Contributor
Contributor

petterreinholdtsen commented Dec 8, 2016

@ccurtsinger

This comment has been minimized.

Show comment
Hide comment
@ccurtsinger

ccurtsinger Dec 9, 2016

Member

Okay, I just pushed a version that has the uncompressed javascript and css sources. If everything looks good on your end, I can tag a release this evening.

Member

ccurtsinger commented Dec 9, 2016

Okay, I just pushed a version that has the uncompressed javascript and css sources. If everything looks good on your end, I can tag a release this evening.

@ccurtsinger ccurtsinger added this to the First release milestone Dec 9, 2016

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Dec 12, 2016

Contributor
Contributor

petterreinholdtsen commented Dec 12, 2016

@ccurtsinger

This comment has been minimized.

Show comment
Hide comment
@ccurtsinger

ccurtsinger Dec 12, 2016

Member

The npm install command will go grab dependencies for the viewer. I'm not terribly comfortable working with npm, so my inclination is to switch to a makefile-based typescript compile instead. That should fit better with the Debian model, where there should be no network dependencies during the build.

Member

ccurtsinger commented Dec 12, 2016

The npm install command will go grab dependencies for the viewer. I'm not terribly comfortable working with npm, so my inclination is to switch to a makefile-based typescript compile instead. That should fit better with the Debian model, where there should be no network dependencies during the build.

@llvilanova

This comment has been minimized.

Show comment
Hide comment
@llvilanova

llvilanova Jul 4, 2017

Contributor

Hi Peter, can any of the tasks of this bug be marked as done?

Contributor

llvilanova commented Jul 4, 2017

Hi Peter, can any of the tasks of this bug be marked as done?

@petterreinholdtsen

This comment has been minimized.

Show comment
Hide comment
@petterreinholdtsen

petterreinholdtsen Jul 4, 2017

Contributor

I can't really speak for Peter, but I had a look and d3-tip and science both seem to be in Debian now. I hope the analysis web pages can be included in the official release tarball, but have not seen any progress there. As for TypeScript, I have no idea what the status is.

Contributor

petterreinholdtsen commented Jul 4, 2017

I can't really speak for Peter, but I had a look and d3-tip and science both seem to be in Debian now. I hope the analysis web pages can be included in the official release tarball, but have not seen any progress there. As for TypeScript, I have no idea what the status is.

@llvilanova

This comment has been minimized.

Show comment
Hide comment
@llvilanova

llvilanova Jul 15, 2017

Contributor

The first point in this task list should be marked as done (the gh-pages branch was moved into the viewer directory).

The second one seems it can be marked as done too; all libraries in the viewer dir seem to have a non-minimised version.

Contributor

llvilanova commented Jul 15, 2017

The first point in this task list should be marked as done (the gh-pages branch was moved into the viewer directory).

The second one seems it can be marked as done too; all libraries in the viewer dir seem to have a non-minimised version.

@emeryberger

This comment has been minimized.

Show comment
Hide comment
@emeryberger

emeryberger Jul 15, 2017

Member

Marked those tasks as done.

Member

emeryberger commented Jul 15, 2017

Marked those tasks as done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment