Rewrite packaging system, rip out Mercurial #63

Merged
merged 19 commits into from Jun 20, 2012

Conversation

Projects
None yet
2 participants
@kini
Member

kini commented Jun 14, 2012

I've considerably simplified the packaging system. The new SPKG is
uncompressed, and has a Mercurial repository tracking spkg-install et
al. like all other SPKGs do. The src/ directory consists of a bunch of
source tarballs of dependencies, an sdist tarball of sagenb itself,
and (optionally) a copy of the git repository of sagenb suitable for
beginning development from.

References to Mercurial, dependencies on Mercurial tools, etc. are also
removed.

kini added some commits Jun 12, 2012

Add some scripts to automate packaging of deps
depdist.sh will be used to create a dist/ directory which should be
moved to the sagenb spkg directory and renamed to src/ .

fetch_deps.py is a helper script for depdist.sh which uses distribute
internals to fetch package tarballs from PyPI.
Fix packaging of git repo
`setup.py sdist` doesn't copy empty directories, as I found out after
a lot of head scratching. This breaks git repos, so we can't trust sdist
to copy our git repo into the source tarball of sagenb. So we'll store
the git repo separately as sagenb.git , and move it into place when
running spkg-install.

@kini kini referenced this pull request Jun 14, 2012

Closed

Jmol enhancements #61

@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 14, 2012

Member

Here's an example of a sagenb SPKG created by dist.sh after these commits: http://wstein.org/home/keshav/files/sagenb-0.9.1.spkg

You will need to take a look at it in order to review this branch because it contains the new SPKG repo I made for sagenb.

Member

kini commented Jun 14, 2012

Here's an example of a sagenb SPKG created by dist.sh after these commits: http://wstein.org/home/keshav/files/sagenb-0.9.1.spkg

You will need to take a look at it in order to review this branch because it contains the new SPKG repo I made for sagenb.

Fix github remote tracking
Apparently you can't set a branch to track a remote tracking branch that
you haven't fetched yet.
@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 14, 2012

Member

SPKG updated according to the latest commit: http://wstein.org/home/keshav/files/sagenb-0.9.1.spkg (same link)

Member

kini commented Jun 14, 2012

SPKG updated according to the latest commit: http://wstein.org/home/keshav/files/sagenb-0.9.1.spkg (same link)

@jhpalmieri

This comment has been minimized.

Show comment
Hide comment
@jhpalmieri

jhpalmieri Jun 15, 2012

Contributor

Would it make any sense to omit the git repository and instead document how to install it for people who want to work on sagenb development?

Contributor

jhpalmieri commented Jun 15, 2012

Would it make any sense to omit the git repository and instead document how to install it for people who want to work on sagenb development?

@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 15, 2012

Member

In my opinion, yes, absolutely. Sage currently ships its repositories inside both source and binary tarballs, which is unorthodox to say the least. The stated goal is to make the barrier of entry to development extremely low in order to gather mathematicians who are not experienced software engineers and open-source development collaborators, but who nonetheless have been hacking on Sage for their own purposes/research, and entice them to give us their code by making it extremely easy to do so. But I don't think this goal makes much sense in the context of sagenb, which is a web app and doesn't really seek contributions from inexperienced programmers who are experts in some other field.

Member

kini commented Jun 15, 2012

In my opinion, yes, absolutely. Sage currently ships its repositories inside both source and binary tarballs, which is unorthodox to say the least. The stated goal is to make the barrier of entry to development extremely low in order to gather mathematicians who are not experienced software engineers and open-source development collaborators, but who nonetheless have been hacking on Sage for their own purposes/research, and entice them to give us their code by making it extremely easy to do so. But I don't think this goal makes much sense in the context of sagenb, which is a web app and doesn't really seek contributions from inexperienced programmers who are experts in some other field.

kini added some commits Jun 15, 2012

Insert old advice from SPKG.txt
This old advice for release management of sagenb was in the SPKG.txt and
was generally out of date. Since the SPKG will now only contain a bunch
of tarballs, a lot of those instructions don't make much sense there, so
I've moved them to the sagenb README for now.
Make packaging the git repo optional
New behavior: `./dist.sh` prepares tarballs of sagenb and its
dependencies in dist/ , and `./dist.sh -g` additionally places an
optimized copy of the git repository in dist/
@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 16, 2012

Member

OK, so packaging the git repo is now optional. Pass -g to dist.sh if you want a git repo.

New SPKG is up at the same URL: http://wstein.org/home/keshav/files/sagenb-0.9.1.spkg

Member

kini commented Jun 16, 2012

OK, so packaging the git repo is now optional. Pass -g to dist.sh if you want a git repo.

New SPKG is up at the same URL: http://wstein.org/home/keshav/files/sagenb-0.9.1.spkg

@jhpalmieri

This comment has been minimized.

Show comment
Hide comment
@jhpalmieri

jhpalmieri Jun 18, 2012

Contributor

When I run dist.sh, I see the following:

Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!

Then everything seems to get downloaded successfully. I also see

warning: sdist: standard file not found: should have one of README, README.txt

I have no idea what this means.

Otherwise, everything looks good, and the new SPKG.txt file has good directions for how to use dist.sh. Should SPKG.txt list the non-included dependencies also, like Python, to record where it should go in SAGE_ROOT/spkg/standard/deps?

Contributor

jhpalmieri commented Jun 18, 2012

When I run dist.sh, I see the following:

Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!

Then everything seems to get downloaded successfully. I also see

warning: sdist: standard file not found: should have one of README, README.txt

I have no idea what this means.

Otherwise, everything looks good, and the new SPKG.txt file has good directions for how to use dist.sh. Should SPKG.txt list the non-included dependencies also, like Python, to record where it should go in SAGE_ROOT/spkg/standard/deps?

@jhpalmieri

This comment has been minimized.

Show comment
Hide comment
@jhpalmieri

jhpalmieri Jun 18, 2012

Contributor

Oh, it might be a good idea to provide instructions (in SPKG.txt? at the top of dist.sh?) for how to use the git repository in the directory sagenb.git, if you run dist.sh -g. (I think you should unpack sagenb-0.9.1.tar.gz and then move sagenb.git to sagenb-0.9.1/.git. Is that right?)

Contributor

jhpalmieri commented Jun 18, 2012

Oh, it might be a good idea to provide instructions (in SPKG.txt? at the top of dist.sh?) for how to use the git repository in the directory sagenb.git, if you run dist.sh -g. (I think you should unpack sagenb-0.9.1.tar.gz and then move sagenb.git to sagenb-0.9.1/.git. Is that right?)

@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 18, 2012

Member

When I run dist.sh, I see the following:

Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!

Those don't appear on a stable connection, such as on sage.math.

Then everything seems to get downloaded successfully. I also see

warning: sdist: standard file not found: should have one of README, README.txt

I have no idea what this means.

To make sure github renders it correctly, our readme is called README.rst, so setuptools complains - this can be ignored (it's only a warning).

Otherwise, everything looks good, and the new SPKG.txt file has good directions for how to use dist.sh. Should SPKG.txt list the non-included dependencies also, like Python, to record where it should go in SAGE_ROOT/spkg/standard/deps?

Good idea.

Member

kini commented Jun 18, 2012

When I run dist.sh, I see the following:

Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!
Download error: [Errno 61] Connection refused -- Some packages may not be found!

Those don't appear on a stable connection, such as on sage.math.

Then everything seems to get downloaded successfully. I also see

warning: sdist: standard file not found: should have one of README, README.txt

I have no idea what this means.

To make sure github renders it correctly, our readme is called README.rst, so setuptools complains - this can be ignored (it's only a warning).

Otherwise, everything looks good, and the new SPKG.txt file has good directions for how to use dist.sh. Should SPKG.txt list the non-included dependencies also, like Python, to record where it should go in SAGE_ROOT/spkg/standard/deps?

Good idea.

@jhpalmieri

This comment has been minimized.

Show comment
Hide comment
@jhpalmieri

jhpalmieri Jun 18, 2012

Contributor

Those don't appear on a stable connection, such as on sage.math.

I just tested, and I also see them on sage.math (although with "Errno 111" instead of "Errno 61"). Also, they are completely reproducible: I see four of these "Download error" messages every time I run the script. Maybe you could add a print statement print "Fetching {0} ...".format(pkg) at the start of the loop, so that the later message "Fetched ..." will reassure the user that the desired package was retrieved successfully.

Contributor

jhpalmieri commented Jun 18, 2012

Those don't appear on a stable connection, such as on sage.math.

I just tested, and I also see them on sage.math (although with "Errno 111" instead of "Errno 61"). Also, they are completely reproducible: I see four of these "Download error" messages every time I run the script. Maybe you could add a print statement print "Fetching {0} ...".format(pkg) at the start of the loop, so that the later message "Fetched ..." will reassure the user that the desired package was retrieved successfully.

@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 19, 2012

Member

Oh, it might be a good idea to provide instructions (in SPKG.txt? at the top of dist.sh?) for how to use the git repository in the directory sagenb.git, if you run dist.sh -g. (I think you should unpack sagenb-0.9.1.tar.gz and then move sagenb.git to sagenb-0.9.1/.git. Is that right?)

No, the instructions are correct - spkg-install will install the git repo correctly if sagenb.git is present in the src directory.

Member

kini commented Jun 19, 2012

Oh, it might be a good idea to provide instructions (in SPKG.txt? at the top of dist.sh?) for how to use the git repository in the directory sagenb.git, if you run dist.sh -g. (I think you should unpack sagenb-0.9.1.tar.gz and then move sagenb.git to sagenb-0.9.1/.git. Is that right?)

No, the instructions are correct - spkg-install will install the git repo correctly if sagenb.git is present in the src directory.

@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 19, 2012

Member

Those don't appear on a stable connection, such as on sage.math.

I just tested, and I also see them on sage.math (although with "Errno 111" instead of "Errno 61"). Also, they are completely reproducible: I see four of these "Download error" messages every time I run the script. Maybe you could add a print statement print "Fetching {0} ...".format(pkg) at the start of the loop, so that the later message "Fetched ..." will reassure the user that the desired package was retrieved successfully.

Good idea. I'll add it when I'm back in front of a computer in a day or so :)

Member

kini commented Jun 19, 2012

Those don't appear on a stable connection, such as on sage.math.

I just tested, and I also see them on sage.math (although with "Errno 111" instead of "Errno 61"). Also, they are completely reproducible: I see four of these "Download error" messages every time I run the script. Maybe you could add a print statement print "Fetching {0} ...".format(pkg) at the start of the loop, so that the later message "Fetched ..." will reassure the user that the desired package was retrieved successfully.

Good idea. I'll add it when I'm back in front of a computer in a day or so :)

@jhpalmieri

This comment has been minimized.

Show comment
Hide comment
@jhpalmieri

jhpalmieri Jun 19, 2012

Contributor

No, the instructions are correct - spkg-install will install the git repo correctly if sagenb.git is present in the src directory.

Oh, right. Sorry.

Contributor

jhpalmieri commented Jun 19, 2012

No, the instructions are correct - spkg-install will install the git repo correctly if sagenb.git is present in the src directory.

Oh, right. Sorry.

@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 20, 2012

Member

OK, I've made the changes you suggested - the ones to the dist scripts in the above commit, and the ones to SPKG.txt in the new SPKG I've just uploaded to the same URL as before: http://wstein.org/home/keshav/files/sagenb-0.9.1.spkg

Member

kini commented Jun 20, 2012

OK, I've made the changes you suggested - the ones to the dist scripts in the above commit, and the ones to SPKG.txt in the new SPKG I've just uploaded to the same URL as before: http://wstein.org/home/keshav/files/sagenb-0.9.1.spkg

@jhpalmieri

This comment has been minimized.

Show comment
Hide comment
@jhpalmieri

jhpalmieri Jun 20, 2012

Contributor

I think this looks good now. I've reviewed the changes and tried out dist.sh. Everything seems to work. Does that suffice for a positive review?

Contributor

jhpalmieri commented Jun 20, 2012

I think this looks good now. I've reviewed the changes and tried out dist.sh. Everything seems to work. Does that suffice for a positive review?

@kini

This comment has been minimized.

Show comment
Hide comment
@kini

kini Jun 20, 2012

Member

Yup! Thanks for helping me out with this!

Member

kini commented Jun 20, 2012

Yup! Thanks for helping me out with this!

kini added a commit that referenced this pull request Jun 20, 2012

Merge pull request #63 from kini/packaging
Rewrite packaging system, rip out Mercurial

@kini kini merged commit b0ce4e7 into sagemath:master Jun 20, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment