New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

packages.json Support for Specifying Compatible ST Versions #291

Closed
wbond opened this Issue Jan 29, 2013 · 70 comments

Comments

@wbond
Owner

wbond commented Jan 29, 2013

With ST3 beta out now, we need to allow users to specify what versions of ST packages work with.

I am inclined to say the only way to support ST3 is for plugin authors to set up a packages.json file. This will mean that all of the authors who use bare github repos will only be available to ST2 users.

I don't think it will be too big of a deal for users who have to port their plugin code to also add a json file to their repo. This will also encourage package author to use semantic versioning instead of date stamps.

Anyone with specific ideas about the specifics of the json for listing compatible versions?

@wbond wbond referenced this issue Jan 29, 2013

Closed

Support sublime 3 #289

@timjrobinson

This comment has been minimized.

Show comment
Hide comment
@timjrobinson

timjrobinson Jan 29, 2013

I'd say structure it as closely as possible to the npm package.json format, many are already familiar with that and it's clear and easy to understand.

timjrobinson commented Jan 29, 2013

I'd say structure it as closely as possible to the npm package.json format, many are already familiar with that and it's clear and easy to understand.

@w00fz

This comment has been minimized.

Show comment
Hide comment
@w00fz

w00fz Jan 30, 2013

I think a package.json is the best approach. Perhaps one suggestion would be to have it handling packages that support different ST versions.
For instance, if you were to follow the npm format, I'd extend the repository attribute to include the ST version dependency.

"repository": { 
    "type" : "git",
    "versions": [{
        ">= 2 < 3":{
            "url" : "http://github.com/user/package"
        },
        "> 3":{
            "url" : "http://github.com/user/package/tree/branch-st3"
        }
    }]
}

Something along those lines, where the versions refer to the sublime versions, could even just be 2 or 3.
This way a developer could maintain different versions of Sublime (usually stable and dev) from the same package.json and would make future ST transitions easier to handle from a Package Control standpoint.

Just an idea.

w00fz commented Jan 30, 2013

I think a package.json is the best approach. Perhaps one suggestion would be to have it handling packages that support different ST versions.
For instance, if you were to follow the npm format, I'd extend the repository attribute to include the ST version dependency.

"repository": { 
    "type" : "git",
    "versions": [{
        ">= 2 < 3":{
            "url" : "http://github.com/user/package"
        },
        "> 3":{
            "url" : "http://github.com/user/package/tree/branch-st3"
        }
    }]
}

Something along those lines, where the versions refer to the sublime versions, could even just be 2 or 3.
This way a developer could maintain different versions of Sublime (usually stable and dev) from the same package.json and would make future ST transitions easier to handle from a Package Control standpoint.

Just an idea.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Jan 30, 2013

Owner

Just in case you guys have not seen it, there is already a packages.json format for Package Control. https://github.com/wbond/sublime_package_control/blob/master/example-packages.json

Owner

wbond commented Jan 30, 2013

Just in case you guys have not seen it, there is already a packages.json format for Package Control. https://github.com/wbond/sublime_package_control/blob/master/example-packages.json

@w00fz

This comment has been minimized.

Show comment
Hide comment
@w00fz

w00fz Jan 30, 2013

Ah sorry.
So what about this?

"platforms": {
    "*": [
        {
            "version": "1.1",
            "client": "st2",
            "url": "http://nodeload.github.com/user/project/archive/v1.1.zip"
        },
        {
            "version": "0.1-dev",
            "client": "st3",
            "url": "http://nodeload.github.com/user/project/archive/branch.zip"
        }
    ]
}

So for each platform type, you have multiple choices that are related to the ST client version and Package Control will know which one to serve to the user. Of course it doesn't have to be st2 or st3 but you get the idea. You could also assume that if the client is missing, it is meant for the current stable version of ST, which should guarantee some sort of backward compatibility for the current status.

w00fz commented Jan 30, 2013

Ah sorry.
So what about this?

"platforms": {
    "*": [
        {
            "version": "1.1",
            "client": "st2",
            "url": "http://nodeload.github.com/user/project/archive/v1.1.zip"
        },
        {
            "version": "0.1-dev",
            "client": "st3",
            "url": "http://nodeload.github.com/user/project/archive/branch.zip"
        }
    ]
}

So for each platform type, you have multiple choices that are related to the ST client version and Package Control will know which one to serve to the user. Of course it doesn't have to be st2 or st3 but you get the idea. You could also assume that if the client is missing, it is meant for the current stable version of ST, which should guarantee some sort of backward compatibility for the current status.

@kblomqvist

This comment has been minimized.

Show comment
Hide comment
@kblomqvist

kblomqvist Jan 30, 2013

Could it be possible to use packages.json to list additional plugins that the plugin is dependent? To get those installed as well if needed.

kblomqvist commented Jan 30, 2013

Could it be possible to use packages.json to list additional plugins that the plugin is dependent? To get those installed as well if needed.

@kostajh

This comment has been minimized.

Show comment
Hide comment
@kostajh

kostajh Jan 30, 2013

Does this mean plugin authors will have to tag new releases and update the packages.json file each time they make an update to a plugin? The existing system of using date stamps for showing updates for plugins is nice for its simplicity.

kostajh commented Jan 30, 2013

Does this mean plugin authors will have to tag new releases and update the packages.json file each time they make an update to a plugin? The existing system of using date stamps for showing updates for plugins is nice for its simplicity.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Jan 30, 2013

Owner

@kostajh That is my current thinking right now.

The only options I see right now are:

  1. Present packages to ST3 users even if they have never been tested. Installing a package will be a crapshoot on ST3.
  2. Try to find someway that through a github repo the developer can indicate if a package works with ST3. The problem here is that I am already consistently hitting the GitHub API rate limits and have been reducing the frequency of updates because of the number of API requests made. Right now each package requires at least 2. Adding a third would make it so I can only updates packages once an hour, but in the near future I wouldn't be able to get all packages in under the hourly rate limit.
  3. Require developers start providing a packages.json file. We could perhaps augment Package Control to make this easier, especially doing such things as bumping revisions.
Owner

wbond commented Jan 30, 2013

@kostajh That is my current thinking right now.

The only options I see right now are:

  1. Present packages to ST3 users even if they have never been tested. Installing a package will be a crapshoot on ST3.
  2. Try to find someway that through a github repo the developer can indicate if a package works with ST3. The problem here is that I am already consistently hitting the GitHub API rate limits and have been reducing the frequency of updates because of the number of API requests made. Right now each package requires at least 2. Adding a third would make it so I can only updates packages once an hour, but in the near future I wouldn't be able to get all packages in under the hourly rate limit.
  3. Require developers start providing a packages.json file. We could perhaps augment Package Control to make this easier, especially doing such things as bumping revisions.
@kostajh

This comment has been minimized.

Show comment
Hide comment
@kostajh

kostajh Jan 30, 2013

That makes sense. Given the options, the third one is the best.

kostajh commented Jan 30, 2013

That makes sense. Given the options, the third one is the best.

@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Jan 30, 2013

Contributor
  1. Is a no-go. What could be done is a setting that makes unmarked packages show and install on ST3 too (like addons in FireFox), this is of course only if you know what you're doing.
  2. What exactly are the two API requests on this? You must be fetching the user, the last modified date (for version) and the description so maybe having the user specify ST3 support in the description would solve it. Other than that it would most probably need some limit raise.
  3. What I think is the most annoying thing about custom JSON is the modified date. I don't even know how it works in combination with the version. Do you even check the date if you check for the version? Or do you update if the version number did not increase but the date did, e.g. for hotfixes? It's probably in the source but I didn't bother to dig into it.
    Other than that, it seems to be the "most secure" way to do things without using hacks (2.) or guesslation (1.). I mean, if you are a package developer you should be known to JSON. And if you update your package for ST3 it should not be that much of a problem to add some additional file with information about it. What does concern me is the amount of updates the repository channel requires. I mean, every plugin that uses the Github or Bitbucket URL implementation has to create a packages.json file and exchange the old entry with the new.

Regarding packages.json formats, I'd probably stick with build numbers since they are really reliable and allow quick adjustment even for small API changes. So, if you specify a minimum and a maximum build number (st3 probably starts at 3000, first build published being 3006) Package Control can also disable packages (with a notice, of course) if a package is not compatible with a recent API change and does not leave the user with a broken plugin.

Contributor

FichteFoll commented Jan 30, 2013

  1. Is a no-go. What could be done is a setting that makes unmarked packages show and install on ST3 too (like addons in FireFox), this is of course only if you know what you're doing.
  2. What exactly are the two API requests on this? You must be fetching the user, the last modified date (for version) and the description so maybe having the user specify ST3 support in the description would solve it. Other than that it would most probably need some limit raise.
  3. What I think is the most annoying thing about custom JSON is the modified date. I don't even know how it works in combination with the version. Do you even check the date if you check for the version? Or do you update if the version number did not increase but the date did, e.g. for hotfixes? It's probably in the source but I didn't bother to dig into it.
    Other than that, it seems to be the "most secure" way to do things without using hacks (2.) or guesslation (1.). I mean, if you are a package developer you should be known to JSON. And if you update your package for ST3 it should not be that much of a problem to add some additional file with information about it. What does concern me is the amount of updates the repository channel requires. I mean, every plugin that uses the Github or Bitbucket URL implementation has to create a packages.json file and exchange the old entry with the new.

Regarding packages.json formats, I'd probably stick with build numbers since they are really reliable and allow quick adjustment even for small API changes. So, if you specify a minimum and a maximum build number (st3 probably starts at 3000, first build published being 3006) Package Control can also disable packages (with a notice, of course) if a package is not compatible with a recent API change and does not leave the user with a broken plugin.

@skuroda

This comment has been minimized.

Show comment
Hide comment
@skuroda

skuroda Jan 31, 2013

Out of curiosity, how would you handle plugins without package.json files? I'm thinking in the context of repositories added by users. Some plugins may be written without the end goal of adding to package control. Now, I know I can clone these into the packages directory, and have package control do updates via git, but sometimes I don't feel like it. In those cases, I add the repository to package control and install normally.

With that being said, I don't think you can rely solely on packages.json files, though it may be as simple as always including user added repos regardless of the presence of that file. Perhaps that's how package control handles it now, I haven't looked. Just thought I would bring this up on the off chance it hadn't been considered already.

Thanks for all the great work.

skuroda commented Jan 31, 2013

Out of curiosity, how would you handle plugins without package.json files? I'm thinking in the context of repositories added by users. Some plugins may be written without the end goal of adding to package control. Now, I know I can clone these into the packages directory, and have package control do updates via git, but sometimes I don't feel like it. In those cases, I add the repository to package control and install normally.

With that being said, I don't think you can rely solely on packages.json files, though it may be as simple as always including user added repos regardless of the presence of that file. Perhaps that's how package control handles it now, I haven't looked. Just thought I would bring this up on the off chance it hadn't been considered already.

Thanks for all the great work.

@jbeales

This comment has been minimized.

Show comment
Hide comment
@jbeales

jbeales Jan 31, 2013

Would it take another API call to see if there's a file called st3-compatible in the repo? That way plugin devs could flag their plugins as st3-compatible, but you could still update based on last-modified dates and devs wouldn't have to create a packages.json if they didn't want to.

jbeales commented Jan 31, 2013

Would it take another API call to see if there's a file called st3-compatible in the repo? That way plugin devs could flag their plugins as st3-compatible, but you could still update based on last-modified dates and devs wouldn't have to create a packages.json if they didn't want to.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Jan 31, 2013

Owner

@jbeales Yeah, that was option 2 above. We would move to about 3000 GitHub api requests per refresh meaning the current 30 minute refresh rate go up to every 60 minutes. Then within 6 months (maybe before ST3 stable is released), we get to 1700 packages and now we are over 5,000 API requests per hour.

By not adding a third API request we can scale up to about 2500 packages not using a packages.json file before we hit the limit. However, by making it a requirement to have a packages.json file for ST3 compatbility, we effectively eliminate new packages that require API requests.

Owner

wbond commented Jan 31, 2013

@jbeales Yeah, that was option 2 above. We would move to about 3000 GitHub api requests per refresh meaning the current 30 minute refresh rate go up to every 60 minutes. Then within 6 months (maybe before ST3 stable is released), we get to 1700 packages and now we are over 5,000 API requests per hour.

By not adding a third API request we can scale up to about 2500 packages not using a packages.json file before we hit the limit. However, by making it a requirement to have a packages.json file for ST3 compatbility, we effectively eliminate new packages that require API requests.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Jan 31, 2013

Owner

In light of the last comment and for consideration moving forward as this scales out, we might need to move to an architecture where the channel server does real git clone/git pull operations on all packages and delivers the .sublime-package files itself.

This would be a bigger undertaking and would require more server resources on my part. Even after gzipping all requests leaving my server, I am serving close to 600GB of JSON a month. If I was delivering package files, I can only imagine I would jump into 2TB+ of bandwidth a month. For that kind of bandwidth we are probably looking at a dedicated server with a $200 a month bill, minimum. The cheapest cloud/VPS bandwidth I've seen is $0.10 a GB. Moving to a dedicated server would make the server more vulnerable to hardware outages.

This architecture would have the benefit of not:

  1. Using GitHub as a fileserver

Ideally it would also have the benefit of simplification of the PC internals since we don't have to extract GitHub zips and remove the extra root folder. However, that would mean users would no longer be able to add custom repos, which is kind of a non-starter.

The downsides are:

  1. Significant ongoing cost
  2. More beefy server/bandwidth, maybe even multiple servers
Owner

wbond commented Jan 31, 2013

In light of the last comment and for consideration moving forward as this scales out, we might need to move to an architecture where the channel server does real git clone/git pull operations on all packages and delivers the .sublime-package files itself.

This would be a bigger undertaking and would require more server resources on my part. Even after gzipping all requests leaving my server, I am serving close to 600GB of JSON a month. If I was delivering package files, I can only imagine I would jump into 2TB+ of bandwidth a month. For that kind of bandwidth we are probably looking at a dedicated server with a $200 a month bill, minimum. The cheapest cloud/VPS bandwidth I've seen is $0.10 a GB. Moving to a dedicated server would make the server more vulnerable to hardware outages.

This architecture would have the benefit of not:

  1. Using GitHub as a fileserver

Ideally it would also have the benefit of simplification of the PC internals since we don't have to extract GitHub zips and remove the extra root folder. However, that would mean users would no longer be able to add custom repos, which is kind of a non-starter.

The downsides are:

  1. Significant ongoing cost
  2. More beefy server/bandwidth, maybe even multiple servers
@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Jan 31, 2013

Owner

@skuroda Yeah, I think the packages.json file would just be for the channel server to present the info. I think if PC connected directly to a GitHub repo, we would list it as available.

Owner

wbond commented Jan 31, 2013

@skuroda Yeah, I think the packages.json file would just be for the channel server to present the info. I think if PC connected directly to a GitHub repo, we would list it as available.

@jbeales

This comment has been minimized.

Show comment
Hide comment
@jbeales

jbeales Jan 31, 2013

If you're already pulling the packages.json file, (or trying to and getting an error back if there isn't one - I haven't looked in the code, and probably wouldn't understand most of the python anyway, so I'm not sure exactly how it works), then would it be an option to include some sort of ST3-compatibility flag in packages.json, but not require anything else, so that devs could put up a packages.json that exists only to flag the plugin as ST3-compatibile?

Regarding the bandwidth/cost/scalability issue, you could also go the other way and move as much as possible to GitHub. I'm not sure how it works now, but what if your server only gave the plugin a list of github repos, then it was up to the plugin to go, look at the repos, and figure out what, if anything can be downloaded &/or updated, (or is this what you're doing now?). This may require that we all have a working copy of git.

You could maybe take that even farther and instead of running your own server, put the list of repos up on github, as a repo itself, (and be pretty meta-tastic). Then you'll serve nothing.

The downside to all this is that the Package Control itself will get a lot more complicated, especially if you do something like bundle git with it.

jbeales commented Jan 31, 2013

If you're already pulling the packages.json file, (or trying to and getting an error back if there isn't one - I haven't looked in the code, and probably wouldn't understand most of the python anyway, so I'm not sure exactly how it works), then would it be an option to include some sort of ST3-compatibility flag in packages.json, but not require anything else, so that devs could put up a packages.json that exists only to flag the plugin as ST3-compatibile?

Regarding the bandwidth/cost/scalability issue, you could also go the other way and move as much as possible to GitHub. I'm not sure how it works now, but what if your server only gave the plugin a list of github repos, then it was up to the plugin to go, look at the repos, and figure out what, if anything can be downloaded &/or updated, (or is this what you're doing now?). This may require that we all have a working copy of git.

You could maybe take that even farther and instead of running your own server, put the list of repos up on github, as a repo itself, (and be pretty meta-tastic). Then you'll serve nothing.

The downside to all this is that the Package Control itself will get a lot more complicated, especially if you do something like bundle git with it.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Jan 31, 2013

Owner

@jbeales Originally PC worked as you describe. However with 1000 packages, it takes 15 minutes to crawl the GitHub API to get info on them all. Plus, as of about 6 months ago, the only way to get 5,000 API requests an hour is to be authenticated. Non-auth'ed users get 60 an hour. So that would users would have to sign up for a GitHub account, and wait 15 minutes before they could pick the package to install. :-)

So right, the initial discussion here is/was about adding something to the packages.json file to specify compatibility. The issue is that probably at most 10% of packages use the packages.json method, and there are some annoyances with it, such as having to tag the repo, update a timestamp, update a version number and update a URL for every release.

The other thing is, I don't look for a packages.json file in each repo. Developers either submit their repo URL, or the URL to a packages.json file. So, again, developer will have to change their workflow to handle that. Currently most developers just make a fix and push it. Then the channel server generates a version number based on the last commit timestamp. Effectively the developers get "free" deployment of all of their packages that way.

Owner

wbond commented Jan 31, 2013

@jbeales Originally PC worked as you describe. However with 1000 packages, it takes 15 minutes to crawl the GitHub API to get info on them all. Plus, as of about 6 months ago, the only way to get 5,000 API requests an hour is to be authenticated. Non-auth'ed users get 60 an hour. So that would users would have to sign up for a GitHub account, and wait 15 minutes before they could pick the package to install. :-)

So right, the initial discussion here is/was about adding something to the packages.json file to specify compatibility. The issue is that probably at most 10% of packages use the packages.json method, and there are some annoyances with it, such as having to tag the repo, update a timestamp, update a version number and update a URL for every release.

The other thing is, I don't look for a packages.json file in each repo. Developers either submit their repo URL, or the URL to a packages.json file. So, again, developer will have to change their workflow to handle that. Currently most developers just make a fix and push it. Then the channel server generates a version number based on the last commit timestamp. Effectively the developers get "free" deployment of all of their packages that way.

@jbeales

This comment has been minimized.

Show comment
Hide comment
@jbeales

jbeales Jan 31, 2013

Aahh, I see the problems. What are you grabbing from github for each plugin? Is there something that could be inspected, somehow, for some sort of a flag, or compatibility number, (like an svn property - does git have those?) If not, I think we're back to square one.

jbeales commented Jan 31, 2013

Aahh, I see the problems. What are you grabbing from github for each plugin? Is there something that could be inspected, somehow, for some sort of a flag, or compatibility number, (like an svn property - does git have those?) If not, I think we're back to square one.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Jan 31, 2013

Owner

I don't believe there is any way to stick it in here: https://api.github.com/repos/wbond/sublime_package_control

Owner

wbond commented Jan 31, 2013

I don't believe there is any way to stick it in here: https://api.github.com/repos/wbond/sublime_package_control

@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Jan 31, 2013

Contributor

I can imagine making the plugin dev able to ommit some keys for packages.json. For example, if you want to proceed with the currect workflow and enable st3 support, you would have to create a file with only the first 4 lines - , + } } from the following example:

{
    "schema_version": "1.3",
    "repositories": {
        "GitHub Example": { "url": "https://github.com/john_smith/github_example", "st_min_version": 3000 },

        "BitBucket Example": {
           "url": "https://bitbucket.org/john_smith/bitbucket_example",
           "st_min_version": 2117, "st_max_version": 2999,
           "platforms": ["windows", "linux"]
        }
    },
    "packages": [
        {
            "name": "Tortoise",
            "description": "Keyboard shortcuts and menu entries to execute TortoiseSVN, TortoiseHg and TortoiseGit commands",
            "author": "Will Bond",
            "homepage": "http://sublime.wbond.net",
            "last_modified": "2011-11-30 22:55:52",
            "platforms": {
                "windows": [
                    {
                        "version": "1.0",
                        "url": "http://sublime.wbond.net/Tortoise.sublime-package"
                    }
                ]   
            }
        }
    ],
    "renamed_packages": {
        "github_example": "Github Example"
    }
}

I think this example pretty much covers all the cases I can imagine (and btw I also removed the "package_name_map"). You could even make the package_control_channel json file accept this format (after quickly converting it) and e.g. let users insert their packages.json files into a "channels": [] list.
By the way, the standard value for "st_max_version" would be 2999 if none of the two was defined. And the "platform" thing in repositories just came to my mind to reduce the necessity to create your own json if this was the only thing you'd need it for.

Let me know what you think.


Edit 13-02-07: Updated the "repositories" so it does not restrict package naming (and looks clearer).

Contributor

FichteFoll commented Jan 31, 2013

I can imagine making the plugin dev able to ommit some keys for packages.json. For example, if you want to proceed with the currect workflow and enable st3 support, you would have to create a file with only the first 4 lines - , + } } from the following example:

{
    "schema_version": "1.3",
    "repositories": {
        "GitHub Example": { "url": "https://github.com/john_smith/github_example", "st_min_version": 3000 },

        "BitBucket Example": {
           "url": "https://bitbucket.org/john_smith/bitbucket_example",
           "st_min_version": 2117, "st_max_version": 2999,
           "platforms": ["windows", "linux"]
        }
    },
    "packages": [
        {
            "name": "Tortoise",
            "description": "Keyboard shortcuts and menu entries to execute TortoiseSVN, TortoiseHg and TortoiseGit commands",
            "author": "Will Bond",
            "homepage": "http://sublime.wbond.net",
            "last_modified": "2011-11-30 22:55:52",
            "platforms": {
                "windows": [
                    {
                        "version": "1.0",
                        "url": "http://sublime.wbond.net/Tortoise.sublime-package"
                    }
                ]   
            }
        }
    ],
    "renamed_packages": {
        "github_example": "Github Example"
    }
}

I think this example pretty much covers all the cases I can imagine (and btw I also removed the "package_name_map"). You could even make the package_control_channel json file accept this format (after quickly converting it) and e.g. let users insert their packages.json files into a "channels": [] list.
By the way, the standard value for "st_max_version" would be 2999 if none of the two was defined. And the "platform" thing in repositories just came to my mind to reduce the necessity to create your own json if this was the only thing you'd need it for.

Let me know what you think.


Edit 13-02-07: Updated the "repositories" so it does not restrict package naming (and looks clearer).

@jbeales

This comment has been minimized.

Show comment
Hide comment
@jbeales

jbeales Jan 31, 2013

I don't see any way to get it into that API call. There would have to be a second call to see if there was a branch called st3 or something.

However, I just discovered, (and this could be a total hack), that if people want to submit ST3-compatible repos, they could change their repo URL to include a querystring, and the API seems to return the same info, just in a slightly different order. See: https://api.github.com/repos/wbond/sublime_package_control?stv=3 This could be a way for devs to manually tag their plugins as ST3-compatible.

Slightly OT again, about the rate limits, it seems like someone should write a github service, (https://github.com/github/github-services#readme), that lets your server know when there's an update. Then when devs set up their repos they should turn on that service, and you won't have to crawl github all the time.

jbeales commented Jan 31, 2013

I don't see any way to get it into that API call. There would have to be a second call to see if there was a branch called st3 or something.

However, I just discovered, (and this could be a total hack), that if people want to submit ST3-compatible repos, they could change their repo URL to include a querystring, and the API seems to return the same info, just in a slightly different order. See: https://api.github.com/repos/wbond/sublime_package_control?stv=3 This could be a way for devs to manually tag their plugins as ST3-compatible.

Slightly OT again, about the rate limits, it seems like someone should write a github service, (https://github.com/github/github-services#readme), that lets your server know when there's an update. Then when devs set up their repos they should turn on that service, and you won't have to crawl github all the time.

@nirix

This comment has been minimized.

Show comment
Hide comment
@nirix

nirix Feb 1, 2013

A GitHub service like @jbeales suggested is a great idea and might be the way to go.

nirix commented Feb 1, 2013

A GitHub service like @jbeales suggested is a great idea and might be the way to go.

@quarnster

This comment has been minimized.

Show comment
Hide comment
@quarnster

quarnster Feb 1, 2013

Contributor

If I was delivering package files, I can only imagine I would jump into 2TB+ of bandwidth a month. For that kind of bandwidth we are probably looking at a dedicated server with a $200 a month bill, minimum. The cheapest cloud/VPS bandwidth I've seen is $0.10 a GB. Moving to a dedicated server would make the server more vulnerable to hardware outages.

Here's a crazy thought: some sort of p2p architecture. I'm on a 100/100mb line with no data use limitations and would happily donate bandwidth and disk space when appropriate. I'm sure there are many others who wouldn't mind participating if it was possible to turn of and/or limit the bandwidth/disk space use.

Contributor

quarnster commented Feb 1, 2013

If I was delivering package files, I can only imagine I would jump into 2TB+ of bandwidth a month. For that kind of bandwidth we are probably looking at a dedicated server with a $200 a month bill, minimum. The cheapest cloud/VPS bandwidth I've seen is $0.10 a GB. Moving to a dedicated server would make the server more vulnerable to hardware outages.

Here's a crazy thought: some sort of p2p architecture. I'm on a 100/100mb line with no data use limitations and would happily donate bandwidth and disk space when appropriate. I'm sure there are many others who wouldn't mind participating if it was possible to turn of and/or limit the bandwidth/disk space use.

@jswartwood

This comment has been minimized.

Show comment
Hide comment
@jswartwood

jswartwood Feb 3, 2013

@wbond do you have any data about how people are using the package definitions currently? Do authors often have a different url per platform?

Here is a draft of a possible new schema that would puts an equal focus on Sublime Text compatibility and platform compatibility. A new releases array could contain an arbitrary set of releases for given compatibility sets. Package Control could take the release with the highest version number that would match the system requirements.

{
  "schema_version": "2.0",
  "packages": [
    {
      "name": "SublimeLinter",
      "description": "Inline lint highlighting for the Sublime Text 2 editor",
      "author": "Kronuz, Aparajita Fishman, Jake Swartwood",
      "homepage": "http://github.com/SublimeLinter/SublimeLinter",
      "releases": [
        {
          "platforms": "*",
          "sublime_text": "*",
          "version": "1.6.12",
          "url": "https://nodeload.github.com/SublimeLinter/SublimeLinter/zip/v1.6.12"
        },
        {
          "platforms": [ "windows", "linux" ],
          "sublime_text": ">= 3",
          "version": "2.0.0",
          "url": "https://nodeload.github.com/SublimeLinter/SublimeLinter/zip/v2.0.0"
        }
      ]
    },
    {
      "name": "SublimeLinter Beta",
      "description": "This version is considered unstable; only install this package if you plan on testing",
      "author": "Kronuz, Aparajita Fishman, Jake Swartwood",
      "homepage": "https://github.com/SublimeLinter/SublimeLinter/tree/beta",
      "releases": [
        {
          "version": "1.7.0-beta.1",
          "url": "https://nodeload.github.com/SublimeLinter/SublimeLinter/zip/beta"
        }
      ]
    }
  ]
}

I would also recommend officially deprecating last_modified in favor of versioning; it always seemed a bit redundant to me.

The sublime_text version number checking should also follow semver for comparison. node-semver has one of the most robust parsers that I've seen (eg. it can do "3.2.x"), but python-semver could be leveraged directly and might be good enough.

jswartwood commented Feb 3, 2013

@wbond do you have any data about how people are using the package definitions currently? Do authors often have a different url per platform?

Here is a draft of a possible new schema that would puts an equal focus on Sublime Text compatibility and platform compatibility. A new releases array could contain an arbitrary set of releases for given compatibility sets. Package Control could take the release with the highest version number that would match the system requirements.

{
  "schema_version": "2.0",
  "packages": [
    {
      "name": "SublimeLinter",
      "description": "Inline lint highlighting for the Sublime Text 2 editor",
      "author": "Kronuz, Aparajita Fishman, Jake Swartwood",
      "homepage": "http://github.com/SublimeLinter/SublimeLinter",
      "releases": [
        {
          "platforms": "*",
          "sublime_text": "*",
          "version": "1.6.12",
          "url": "https://nodeload.github.com/SublimeLinter/SublimeLinter/zip/v1.6.12"
        },
        {
          "platforms": [ "windows", "linux" ],
          "sublime_text": ">= 3",
          "version": "2.0.0",
          "url": "https://nodeload.github.com/SublimeLinter/SublimeLinter/zip/v2.0.0"
        }
      ]
    },
    {
      "name": "SublimeLinter Beta",
      "description": "This version is considered unstable; only install this package if you plan on testing",
      "author": "Kronuz, Aparajita Fishman, Jake Swartwood",
      "homepage": "https://github.com/SublimeLinter/SublimeLinter/tree/beta",
      "releases": [
        {
          "version": "1.7.0-beta.1",
          "url": "https://nodeload.github.com/SublimeLinter/SublimeLinter/zip/beta"
        }
      ]
    }
  ]
}

I would also recommend officially deprecating last_modified in favor of versioning; it always seemed a bit redundant to me.

The sublime_text version number checking should also follow semver for comparison. node-semver has one of the most robust parsers that I've seen (eg. it can do "3.2.x"), but python-semver could be leveraged directly and might be good enough.

@jlegewie

This comment has been minimized.

Show comment
Hide comment
@jlegewie

jlegewie Feb 4, 2013

Even when you manage to make it work with the current trafic, it should be future proof. Did you try to request a higher rate for GitHub's API? If that creates additional monthly costs, I am all for getting it financed by the community (maybe a way to make a monthly $2 contribution or something) or maybe Jon is willing to jump in.

If deployment is getting more complicated for the package developers, I would also suggest some Submit Plugin to Package Control command in ST3 that takes care of a lot of the work (e.g. generates a packages.json etc).

jlegewie commented Feb 4, 2013

Even when you manage to make it work with the current trafic, it should be future proof. Did you try to request a higher rate for GitHub's API? If that creates additional monthly costs, I am all for getting it financed by the community (maybe a way to make a monthly $2 contribution or something) or maybe Jon is willing to jump in.

If deployment is getting more complicated for the package developers, I would also suggest some Submit Plugin to Package Control command in ST3 that takes care of a lot of the work (e.g. generates a packages.json etc).

@zchrykng

This comment has been minimized.

Show comment
Hide comment
@zchrykng

zchrykng Feb 7, 2013

I was wondering how homebrew manages to handle their load. I believe everything in run through github, but the only data that is retrieved until actual install would be a equivalent to the package.json file. I think it would be possible for PC to run the same way.

zchrykng commented Feb 7, 2013

I was wondering how homebrew manages to handle their load. I believe everything in run through github, but the only data that is retrieved until actual install would be a equivalent to the package.json file. I think it would be possible for PC to run the same way.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Feb 7, 2013

Owner

@zchrykng

Well, homebrew is quite different:

  1. It only runs on OS X
  2. It uses Git for everything
  3. All package metadata is centralized in a git repo
  4. Someone has to write an install formula for every package that is included

Currently, the only thing that is served from my server is the repositories.json file. GitHub isn't really in the business of acting as a CDN to serve a static file 10M+ times a month.

The real solution to the problem is to set up a github service. That, however, probably takes at least as much work for each package developer as creating a packages.json file. I've created #329 to continue any discussion related to that.

Owner

wbond commented Feb 7, 2013

@zchrykng

Well, homebrew is quite different:

  1. It only runs on OS X
  2. It uses Git for everything
  3. All package metadata is centralized in a git repo
  4. Someone has to write an install formula for every package that is included

Currently, the only thing that is served from my server is the repositories.json file. GitHub isn't really in the business of acting as a CDN to serve a static file 10M+ times a month.

The real solution to the problem is to set up a github service. That, however, probably takes at least as much work for each package developer as creating a packages.json file. I've created #329 to continue any discussion related to that.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Feb 7, 2013

Owner

@jswartwood

last_modified is used for the community package list webpage, so I think I'll keep it for now. Although I don't have any analytics about how many people view that sorting on the page.

Thanks for putting that proposal together. I am going to think about that some, but I like the idea of having a list of compatible platforms. The other concern is the update to the 2.0 schema and old versions of package control. Unfortunately it seems that a decent number of people run old versions. Perhaps we can update the packages.json to 2.0, but have the channel server translate it into a 1.3 version that would be backwards compatible with older versions of PC.

semver.py is currently bundled, but I need to fix a bug with comparing semantic versions with date-based versions.

Owner

wbond commented Feb 7, 2013

@jswartwood

last_modified is used for the community package list webpage, so I think I'll keep it for now. Although I don't have any analytics about how many people view that sorting on the page.

Thanks for putting that proposal together. I am going to think about that some, but I like the idea of having a list of compatible platforms. The other concern is the update to the 2.0 schema and old versions of package control. Unfortunately it seems that a decent number of people run old versions. Perhaps we can update the packages.json to 2.0, but have the channel server translate it into a 1.3 version that would be backwards compatible with older versions of PC.

semver.py is currently bundled, but I need to fix a bug with comparing semantic versions with date-based versions.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Feb 25, 2013

Owner

Hmm, so it seems there is a bit of discussion around dependencies. What do you guys think about holding that off until version 2.1? I don't think it would introduce any backwards compatibility issues.

In terms of the structure of the channel file, I imagine it looking like:

{
    "schema_version": "2.0",
    "repositories": [
        "https://sublime.wbond.net/community_repository",
        "https://sublime.wbond.net/packages.json",
        "http://wuub.net/packages.json",
        "https://raw.github.com/afterdesign/MacTerminal/master/packages.json",
        "https://raw.github.com/ajryan/CSharpreter/master/packages.json",
        "https://raw.github.com/aparajita/active4d-sublime/master/package_control.json",
        "https://raw.github.com/aponxi/sublime-better-coffeescript/master/package.json",
        "https://raw.github.com/apophys/sublime-packages/master/packages.json",
        "https://raw.github.com/blueplanet/sublime-text-2-octopress/master/packages.json",
        "https://raw.github.com/camperdave/EC2Upload/master/packages.json",
        "https://raw.github.com/ccpalettes/sublime-lorem-text/master/packages.json",
        "https://raw.github.com/chancedai/sublime-cross/master/packages.json",
        "https://raw.github.com/chikatoike/IMESupport/master/packages.json",
        "https://raw.github.com/chriswong/sublime-mootools-snippets/master/packages.json",
        "https://raw.github.com/colinta/sublime_packages/master/packages.json",
        "https://raw.github.com/connec/Open-in-ConEmu/master/packages.json",
        "https://raw.github.com/corbinian/GrowlNotifier/master/packages.json",
        "https://raw.github.com/csytan/sublime-text-2-github/master/packages.json",
        "https://raw.github.com/dallbee/RandomText/master/packages.json",
        "https://raw.github.com/damccull/sublimetext2-SolarizedToggle/master/packages.json",
        "https://raw.github.com/darinmorrison/sublime-packages/master/packages.json",
        "https://raw.github.com/dexbol/sublime-TortoiseSVN/master/packages.json",
        "https://raw.github.com/enriquein/JavaSetterGetter/master/packages.json",
        "https://raw.github.com/fabric-engine/Sublime-KL/master/packages.json",
        "https://raw.github.com/farcaller/DashDoc/master/packages.json",
        "https://raw.github.com/FichteFoll/sublime_packages/master/package_control.json",
        "https://raw.github.com/filcab/SublimeLLDB/master/packages.json",
        "https://raw.github.com/Floobits/sublime-text-2-plugin/master/packages.json",
        "https://raw.github.com/francodacosta/sublime-php-getters-setters/master/packages.json",
        "https://raw.github.com/freewizard/sublime_packages/master/package_control.json",
        "https://raw.github.com/gcollazo/sublime_packages/master/packages.json",
        "https://raw.github.com/Harrison-M/indent.txt-sublime/master/packages.json",
        "https://raw.github.com/himynameisjonas/sublime-ia-writer-opener/master/packages.json",
        "https://raw.github.com/ikeike443/Sublime-Scalariform/master/packages.json",
        "https://raw.github.com/jadb/st2-search-cakephp-api/master/packages.json",
        "https://raw.github.com/jadb/st2-search-cakephp-book/master/packages.json",
        "https://raw.github.com/jeffturcotte/sublime_packages/master/packages.json",
        "https://raw.github.com/jfernandez/SublimeRailsEval/master/packages.json",
        "https://raw.github.com/jfromaniello/sublime-unity-recents/master/packages.json",
        "https://raw.github.com/joeyespo/sublimetext-console-exec/master/packages.json",
        "https://raw.github.com/jpswelch/sublime-google-tasks/master/packages.json",
        "https://raw.github.com/kairyou/SublimeTmpl/master/packages.json",
        "https://raw.github.com/Kaizhi/SublimeUpdater/master/packages.json",
        "https://raw.github.com/kallepersson/Sublime-Finder/master/packages.json",
        "https://raw.github.com/keeganstreet/sublime-elfinder/master/packages.json",
        "https://raw.github.com/keeganstreet/sublime-specificity/master/packages.json",
        "https://raw.github.com/Kronuz/SublimeCodeIntel/master/package_control.json",
        "https://raw.github.com/kylederkacz/lettuce-farmer/master/packages.json",
        "https://raw.github.com/Learning/sublime_packages/master/packages.json",
        "https://raw.github.com/leporo/SublimeYammy/master/packages.json",
        "https://raw.github.com/LONGMANi/sublimetext-autobackups/master/packages.json",
        "https://raw.github.com/LONGMANi/sublimetext-stringutilities/master/packages.json",
        "https://raw.github.com/lucifr/CNPunctuationAutopair/master/packages.json",
        "https://raw.github.com/lyapun/sublime-text-2-python-test-runner/master/packages.json",
        "https://raw.github.com/mablo/sublime-text-2-meld-diff/master/packages.json",
        "https://raw.github.com/Makopo/sublime-text-lsl/master/package.json",
        "https://raw.github.com/mekwall/obsidian-color-scheme/master/packages.json",
        "https://raw.github.com/merlinthered/sublime-rainmeter/master/packages.json",
        "https://raw.github.com/mischah/Console-API-Snippets/master/packages.json",
        "https://raw.github.com/Mozillion/SublimeSpeech/master/packages.json",
        "https://raw.github.com/naomichi-y/php_syntax_checker/master/packages.json",
        "https://raw.github.com/naomichi-y/string_counter/master/packages.json",
        "https://raw.github.com/NicholasBuse/sublime_packages/master/packages.json",
        "https://raw.github.com/nLight/Phing/master/packages.json",
        "https://raw.github.com/quarnster/CompleteSharp/master/package.json",
        "https://raw.github.com/quarnster/SublimeClang/master/package.json",
        "https://raw.github.com/quarnster/SublimeJava/master/package.json",
        "https://raw.github.com/raymondkww/R-sublime/master/packages.json",
        "https://raw.github.com/rbo/st2-perltidy/master/package_control.json",
        "https://raw.github.com/relikd/plist-binary_sublime/master/packages.json",
        "https://raw.github.com/robcowie/SublimeTODO/master/packages.json",
        "https://raw.github.com/ronmichael/sublime-sparql/master/packages.json",
        "https://raw.github.com/royisme/SublimeOctopressTool/master/packages.json",
        "https://raw.github.com/rpowers/sublime_stata/master/packages.json",
        "https://raw.github.com/sdolard/sublime-jsrevival/master/packages.json",
        "https://raw.github.com/seanliang/ConvertToUTF8/master/packages.json",
        "https://raw.github.com/sentience/DokuWiki/master/packages.json",
        "https://raw.github.com/sentience/HyperlinkHelper/master/packages.json",
        "https://raw.github.com/shivkumarganesh/VisSub/master/packages.json",
        "https://raw.github.com/sindresorhus/sublime-jsrun/master/packages.json",
        "https://raw.github.com/SublimeLinter/SublimeLinter/master/package_control.json",
        "https://raw.github.com/svaiter/SublimeJEDI/master/packages.json",
        "https://raw.github.com/tbfisher/sublimetext-Pandoc/master/packages.json",
        "https://raw.github.com/tcrosen/recessify/master/packages.json",
        "https://raw.github.com/Ted-Mohamed/Split/master/packages.json",
        "https://raw.github.com/theadamlt/sublime_packages/master/packages.json",
        "https://raw.github.com/tiger2wander/sublime_packages/master/packages.json",
        "https://raw.github.com/tillig/SublimeMSBuild/master/packages.json",
        "https://raw.github.com/timonwong/sublime_packages/master/packages.json",
        "https://raw.github.com/tmanderson/VintageLines/master/packages.json",
        "https://raw.github.com/tomascayuelas/coolcodescheme/master/packages.json",
        "https://raw.github.com/tsteur/sublimetext-tita/master/packages.json",
        "https://raw.github.com/ttscoff/MarkdownEditing/master/packages.json",
        "https://raw.github.com/tzvetkoff/sublime_stupid_indent/master/packages.json",
        "https://raw.github.com/vkocubinsky/sublime_packages/master/packages.json",
        "https://raw.github.com/wallysalami/QuickLook/master/packages.json",
        "https://raw.github.com/weslly/sublime_packages/master/packages.json",
        "https://raw.github.com/WhatWeDo/Sublime-Text-2-Compass-Build-System/master/packages.json",
        "https://raw.github.com/wistful/SublimeAutoPEP8/master/packages.json",
        "https://raw.github.com/wukkuan/AMD-Module-Editor/master/packages.json",
        "https://raw.github.com/xgenvn/InputHelper/master/packages.json",
        "https://raw.github.com/yangsu/sublime-io/master/packages.json",
        "https://raw.github.com/yangsu/sublime-octopress/master/packages.json",
        "https://raw.github.com/yangsu/sublime-vhdl/master/packages.json",
        "https://raw.github.com/zfkun/sublime-kissy-snippets/master/packages.json"
    ],
    "packages_cache": {
        "https://sublime.wbond.net/community_repository": [
            {
                "name": "MinimalFortran",
                ...
            },
            ...
        ]
    }
}

The packages_cache entry would be optional and would contain fully-detailed packages entries for one or more repositories. By fully-detailed, I mean that the details key would be invalid. Instead, a crawler/client would need to hit the API of GitHub/BitBucket to provide the releases, description, etc.

PC 2.0 would save this in a new setting called channels (which will replace the existing repository_channels setting):

{
    "channels": ["https://sublime.wbond.net/default_channel"]
}

Based on some of the discussion here, I think the community_repository would look like:

{
    "schema_version": "2.0",
    "packages": [
        {
            "name": "MinimalFortran",
            "details": "https://github.com/315234/MinimalFortran"
        },
        {
            "name": "Live CSS",
            "details": "https://github.com/a-sk/livecss"
        },
        {
            "name": "Build on Save",
            "details": "https://github.com/a-sk/sublime-build-on-save"
        },
        {
            "name": "AndyJS2",
            "details": "https://github.com/agibsonsw/AndyJS2",
            "previous_names": ["AndyJS"]
        }
    ]
}

So https://github.com/wbond/package_control_channel would now contain two json files. My "crawler" would do the work of grabbing all of the data from the details URLs and pushing it into the packages_cache. This should be done via webhooks.

It will also be necessary to enhance the various providers in PC 2.0 to be able to grab the necessary details for users who are not running a caching channel server.

Due to the changes in the way that channels and repositories are becoming more well-formed, we will need to change how custom repositories are added. In the Package Control.sublime-settings file, all https://github.com/username/repo, https://github.com/username and https://bitbucket.org/username/repo entries in the repositories key will be migrated to packages. All other repositories will be left the same:

{
    "repositories": [
        "https://non-github-bitbucket.com/repo.json"
    ],
    "packages": [
        {
            "details": "https://github.com/username/repo"
        },
        {
            "details": "https://bitbucket.org/username/repo"
        },
        {
            "name": "Custom Name",
            "details": "https://github.com/username/repo"
        }
    ]
}

Now, should there be a new command Add Package for GitHub/BitBucket URLs? Or should we leave it just Add Repository and have the code determine if it is a supported GitHub/BitBucket URL?

Owner

wbond commented Feb 25, 2013

Hmm, so it seems there is a bit of discussion around dependencies. What do you guys think about holding that off until version 2.1? I don't think it would introduce any backwards compatibility issues.

In terms of the structure of the channel file, I imagine it looking like:

{
    "schema_version": "2.0",
    "repositories": [
        "https://sublime.wbond.net/community_repository",
        "https://sublime.wbond.net/packages.json",
        "http://wuub.net/packages.json",
        "https://raw.github.com/afterdesign/MacTerminal/master/packages.json",
        "https://raw.github.com/ajryan/CSharpreter/master/packages.json",
        "https://raw.github.com/aparajita/active4d-sublime/master/package_control.json",
        "https://raw.github.com/aponxi/sublime-better-coffeescript/master/package.json",
        "https://raw.github.com/apophys/sublime-packages/master/packages.json",
        "https://raw.github.com/blueplanet/sublime-text-2-octopress/master/packages.json",
        "https://raw.github.com/camperdave/EC2Upload/master/packages.json",
        "https://raw.github.com/ccpalettes/sublime-lorem-text/master/packages.json",
        "https://raw.github.com/chancedai/sublime-cross/master/packages.json",
        "https://raw.github.com/chikatoike/IMESupport/master/packages.json",
        "https://raw.github.com/chriswong/sublime-mootools-snippets/master/packages.json",
        "https://raw.github.com/colinta/sublime_packages/master/packages.json",
        "https://raw.github.com/connec/Open-in-ConEmu/master/packages.json",
        "https://raw.github.com/corbinian/GrowlNotifier/master/packages.json",
        "https://raw.github.com/csytan/sublime-text-2-github/master/packages.json",
        "https://raw.github.com/dallbee/RandomText/master/packages.json",
        "https://raw.github.com/damccull/sublimetext2-SolarizedToggle/master/packages.json",
        "https://raw.github.com/darinmorrison/sublime-packages/master/packages.json",
        "https://raw.github.com/dexbol/sublime-TortoiseSVN/master/packages.json",
        "https://raw.github.com/enriquein/JavaSetterGetter/master/packages.json",
        "https://raw.github.com/fabric-engine/Sublime-KL/master/packages.json",
        "https://raw.github.com/farcaller/DashDoc/master/packages.json",
        "https://raw.github.com/FichteFoll/sublime_packages/master/package_control.json",
        "https://raw.github.com/filcab/SublimeLLDB/master/packages.json",
        "https://raw.github.com/Floobits/sublime-text-2-plugin/master/packages.json",
        "https://raw.github.com/francodacosta/sublime-php-getters-setters/master/packages.json",
        "https://raw.github.com/freewizard/sublime_packages/master/package_control.json",
        "https://raw.github.com/gcollazo/sublime_packages/master/packages.json",
        "https://raw.github.com/Harrison-M/indent.txt-sublime/master/packages.json",
        "https://raw.github.com/himynameisjonas/sublime-ia-writer-opener/master/packages.json",
        "https://raw.github.com/ikeike443/Sublime-Scalariform/master/packages.json",
        "https://raw.github.com/jadb/st2-search-cakephp-api/master/packages.json",
        "https://raw.github.com/jadb/st2-search-cakephp-book/master/packages.json",
        "https://raw.github.com/jeffturcotte/sublime_packages/master/packages.json",
        "https://raw.github.com/jfernandez/SublimeRailsEval/master/packages.json",
        "https://raw.github.com/jfromaniello/sublime-unity-recents/master/packages.json",
        "https://raw.github.com/joeyespo/sublimetext-console-exec/master/packages.json",
        "https://raw.github.com/jpswelch/sublime-google-tasks/master/packages.json",
        "https://raw.github.com/kairyou/SublimeTmpl/master/packages.json",
        "https://raw.github.com/Kaizhi/SublimeUpdater/master/packages.json",
        "https://raw.github.com/kallepersson/Sublime-Finder/master/packages.json",
        "https://raw.github.com/keeganstreet/sublime-elfinder/master/packages.json",
        "https://raw.github.com/keeganstreet/sublime-specificity/master/packages.json",
        "https://raw.github.com/Kronuz/SublimeCodeIntel/master/package_control.json",
        "https://raw.github.com/kylederkacz/lettuce-farmer/master/packages.json",
        "https://raw.github.com/Learning/sublime_packages/master/packages.json",
        "https://raw.github.com/leporo/SublimeYammy/master/packages.json",
        "https://raw.github.com/LONGMANi/sublimetext-autobackups/master/packages.json",
        "https://raw.github.com/LONGMANi/sublimetext-stringutilities/master/packages.json",
        "https://raw.github.com/lucifr/CNPunctuationAutopair/master/packages.json",
        "https://raw.github.com/lyapun/sublime-text-2-python-test-runner/master/packages.json",
        "https://raw.github.com/mablo/sublime-text-2-meld-diff/master/packages.json",
        "https://raw.github.com/Makopo/sublime-text-lsl/master/package.json",
        "https://raw.github.com/mekwall/obsidian-color-scheme/master/packages.json",
        "https://raw.github.com/merlinthered/sublime-rainmeter/master/packages.json",
        "https://raw.github.com/mischah/Console-API-Snippets/master/packages.json",
        "https://raw.github.com/Mozillion/SublimeSpeech/master/packages.json",
        "https://raw.github.com/naomichi-y/php_syntax_checker/master/packages.json",
        "https://raw.github.com/naomichi-y/string_counter/master/packages.json",
        "https://raw.github.com/NicholasBuse/sublime_packages/master/packages.json",
        "https://raw.github.com/nLight/Phing/master/packages.json",
        "https://raw.github.com/quarnster/CompleteSharp/master/package.json",
        "https://raw.github.com/quarnster/SublimeClang/master/package.json",
        "https://raw.github.com/quarnster/SublimeJava/master/package.json",
        "https://raw.github.com/raymondkww/R-sublime/master/packages.json",
        "https://raw.github.com/rbo/st2-perltidy/master/package_control.json",
        "https://raw.github.com/relikd/plist-binary_sublime/master/packages.json",
        "https://raw.github.com/robcowie/SublimeTODO/master/packages.json",
        "https://raw.github.com/ronmichael/sublime-sparql/master/packages.json",
        "https://raw.github.com/royisme/SublimeOctopressTool/master/packages.json",
        "https://raw.github.com/rpowers/sublime_stata/master/packages.json",
        "https://raw.github.com/sdolard/sublime-jsrevival/master/packages.json",
        "https://raw.github.com/seanliang/ConvertToUTF8/master/packages.json",
        "https://raw.github.com/sentience/DokuWiki/master/packages.json",
        "https://raw.github.com/sentience/HyperlinkHelper/master/packages.json",
        "https://raw.github.com/shivkumarganesh/VisSub/master/packages.json",
        "https://raw.github.com/sindresorhus/sublime-jsrun/master/packages.json",
        "https://raw.github.com/SublimeLinter/SublimeLinter/master/package_control.json",
        "https://raw.github.com/svaiter/SublimeJEDI/master/packages.json",
        "https://raw.github.com/tbfisher/sublimetext-Pandoc/master/packages.json",
        "https://raw.github.com/tcrosen/recessify/master/packages.json",
        "https://raw.github.com/Ted-Mohamed/Split/master/packages.json",
        "https://raw.github.com/theadamlt/sublime_packages/master/packages.json",
        "https://raw.github.com/tiger2wander/sublime_packages/master/packages.json",
        "https://raw.github.com/tillig/SublimeMSBuild/master/packages.json",
        "https://raw.github.com/timonwong/sublime_packages/master/packages.json",
        "https://raw.github.com/tmanderson/VintageLines/master/packages.json",
        "https://raw.github.com/tomascayuelas/coolcodescheme/master/packages.json",
        "https://raw.github.com/tsteur/sublimetext-tita/master/packages.json",
        "https://raw.github.com/ttscoff/MarkdownEditing/master/packages.json",
        "https://raw.github.com/tzvetkoff/sublime_stupid_indent/master/packages.json",
        "https://raw.github.com/vkocubinsky/sublime_packages/master/packages.json",
        "https://raw.github.com/wallysalami/QuickLook/master/packages.json",
        "https://raw.github.com/weslly/sublime_packages/master/packages.json",
        "https://raw.github.com/WhatWeDo/Sublime-Text-2-Compass-Build-System/master/packages.json",
        "https://raw.github.com/wistful/SublimeAutoPEP8/master/packages.json",
        "https://raw.github.com/wukkuan/AMD-Module-Editor/master/packages.json",
        "https://raw.github.com/xgenvn/InputHelper/master/packages.json",
        "https://raw.github.com/yangsu/sublime-io/master/packages.json",
        "https://raw.github.com/yangsu/sublime-octopress/master/packages.json",
        "https://raw.github.com/yangsu/sublime-vhdl/master/packages.json",
        "https://raw.github.com/zfkun/sublime-kissy-snippets/master/packages.json"
    ],
    "packages_cache": {
        "https://sublime.wbond.net/community_repository": [
            {
                "name": "MinimalFortran",
                ...
            },
            ...
        ]
    }
}

The packages_cache entry would be optional and would contain fully-detailed packages entries for one or more repositories. By fully-detailed, I mean that the details key would be invalid. Instead, a crawler/client would need to hit the API of GitHub/BitBucket to provide the releases, description, etc.

PC 2.0 would save this in a new setting called channels (which will replace the existing repository_channels setting):

{
    "channels": ["https://sublime.wbond.net/default_channel"]
}

Based on some of the discussion here, I think the community_repository would look like:

{
    "schema_version": "2.0",
    "packages": [
        {
            "name": "MinimalFortran",
            "details": "https://github.com/315234/MinimalFortran"
        },
        {
            "name": "Live CSS",
            "details": "https://github.com/a-sk/livecss"
        },
        {
            "name": "Build on Save",
            "details": "https://github.com/a-sk/sublime-build-on-save"
        },
        {
            "name": "AndyJS2",
            "details": "https://github.com/agibsonsw/AndyJS2",
            "previous_names": ["AndyJS"]
        }
    ]
}

So https://github.com/wbond/package_control_channel would now contain two json files. My "crawler" would do the work of grabbing all of the data from the details URLs and pushing it into the packages_cache. This should be done via webhooks.

It will also be necessary to enhance the various providers in PC 2.0 to be able to grab the necessary details for users who are not running a caching channel server.

Due to the changes in the way that channels and repositories are becoming more well-formed, we will need to change how custom repositories are added. In the Package Control.sublime-settings file, all https://github.com/username/repo, https://github.com/username and https://bitbucket.org/username/repo entries in the repositories key will be migrated to packages. All other repositories will be left the same:

{
    "repositories": [
        "https://non-github-bitbucket.com/repo.json"
    ],
    "packages": [
        {
            "details": "https://github.com/username/repo"
        },
        {
            "details": "https://bitbucket.org/username/repo"
        },
        {
            "name": "Custom Name",
            "details": "https://github.com/username/repo"
        }
    ]
}

Now, should there be a new command Add Package for GitHub/BitBucket URLs? Or should we leave it just Add Repository and have the code determine if it is a supported GitHub/BitBucket URL?

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Feb 25, 2013

Owner

Also, thank you for everyone's contributions so far. I think we are getting real close to a setup that makes it easy to:

  1. Understand
  2. Promote good workflow (tags, platform compatibility)
  3. Easy to maintain
Owner

wbond commented Feb 25, 2013

Also, thank you for everyone's contributions so far. I think we are getting real close to a setup that makes it easy to:

  1. Understand
  2. Promote good workflow (tags, platform compatibility)
  3. Easy to maintain
@zchrykng

This comment has been minimized.

Show comment
Hide comment
@zchrykng

zchrykng Feb 25, 2013

Now, should there be a new command Add Package for GitHub/BitBucket URLs? Or should we leave it just Add Repository and have the code determine if it is a supported GitHub/BitBucket URL?

@wbond I would say that personally I would prefer the separate commands, but I could see that being confusing to some less technical users.

Also, I have completed a basic implementation of a SemVer object, if people think it would be helpful in anyway. And if it if not helpful, I would appreciate any feedback , since this is the first class I have built in Python.

zchrykng commented Feb 25, 2013

Now, should there be a new command Add Package for GitHub/BitBucket URLs? Or should we leave it just Add Repository and have the code determine if it is a supported GitHub/BitBucket URL?

@wbond I would say that personally I would prefer the separate commands, but I could see that being confusing to some less technical users.

Also, I have completed a basic implementation of a SemVer object, if people think it would be helpful in anyway. And if it if not helpful, I would appreciate any feedback , since this is the first class I have built in Python.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Feb 25, 2013

Owner

@zchrykng I've opened #355 to track the feature required by PC 2.0 for version comparison. I also dropped in a link to the current semver library PC uses in case that is useful in your work.

We are also going to need to build a small function that converts ST build numbers to SemVer strings, to get this to work properly.

Owner

wbond commented Feb 25, 2013

@zchrykng I've opened #355 to track the feature required by PC 2.0 for version comparison. I also dropped in a link to the current semver library PC uses in case that is useful in your work.

We are also going to need to build a small function that converts ST build numbers to SemVer strings, to get this to work properly.

@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Feb 25, 2013

Contributor

I'd also go for different commands, it's less confusing imo. You should just clarify what "repositories" and "packages" are, e.g. on the Package Control installation site, and it should be clear.

Contributor

FichteFoll commented Feb 25, 2013

I'd also go for different commands, it's less confusing imo. You should just clarify what "repositories" and "packages" are, e.g. on the Package Control installation site, and it should be clear.

@jisaacks

This comment has been minimized.

Show comment
Hide comment
@jisaacks

jisaacks Feb 27, 2013

I think there should be a setting to specify to unpack the plugin.

The icons in GitGutter will not load otherwise.

See package control issue: #349

and GitGutter issue 46

jisaacks commented Feb 27, 2013

I think there should be a setting to specify to unpack the plugin.

The icons in GitGutter will not load otherwise.

See package control issue: #349

and GitGutter issue 46

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Feb 27, 2013

Owner

@jisaacks Issue #349 is where the discussion about packages vs loose files is happening.

Owner

wbond commented Feb 27, 2013

@jisaacks Issue #349 is where the discussion about packages vs loose files is happening.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Feb 27, 2013

Owner

Sorry for the mixed signals on #349 about directing people to this issue. I am now inclined to have a magic file in the repo prevent packaging so that users who install via Add Repository (either by choice or installing a fork/branch) won't have issues either.

Owner

wbond commented Feb 27, 2013

Sorry for the mixed signals on #349 about directing people to this issue. I am now inclined to have a magic file in the repo prevent packaging so that users who install via Add Repository (either by choice or installing a fork/branch) won't have issues either.

@jisaacks

This comment has been minimized.

Show comment
Hide comment
@jisaacks

jisaacks commented Feb 27, 2013

@wbond ok got it.

@schlamar

This comment has been minimized.

Show comment
Hide comment
@schlamar

schlamar Mar 11, 2013

Contributor

@wbond I would suggest that you add an archs identifier additionally to platforms. This would really help packages with binary requirements. For example the fictional package SublimePyv8 would install the correct .pyd/.so depending on the current arch/platform and packages requiring it could just do from SublimePyv8 import pyv8 without any platform/arch check.

This is already a nice feature for the current version and with #166 implemented this would be more than great.

Contributor

schlamar commented Mar 11, 2013

@wbond I would suggest that you add an archs identifier additionally to platforms. This would really help packages with binary requirements. For example the fictional package SublimePyv8 would install the correct .pyd/.so depending on the current arch/platform and packages requiring it could just do from SublimePyv8 import pyv8 without any platform/arch check.

This is already a nice feature for the current version and with #166 implemented this would be more than great.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Mar 11, 2013

Owner

The arch has been supported in the platform spec for a while. You can see it in master at https://github.com/wbond/sublime_package_control/blob/master/package_control/providers/platform_comparator.py#L9.

Owner

wbond commented Mar 11, 2013

The arch has been supported in the platform spec for a while. You can see it in master at https://github.com/wbond/sublime_package_control/blob/master/package_control/providers/platform_comparator.py#L9.

@schlamar

This comment has been minimized.

Show comment
Hide comment
@schlamar

schlamar Mar 11, 2013

Contributor

Ah, great. 👍 Any ETA for the ST3 format?

Contributor

schlamar commented Mar 11, 2013

Ah, great. 👍 Any ETA for the ST3 format?

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Mar 11, 2013

Owner

I have it implemented locally, but now I am working on the channel server code so I can test it all. Hopefully I'll have something pushed up in the next few days.

I want to open source the website code, and share code between PC and the channel server, so now I am getting up to speed on Python 3 web development. Once the site is open-sourced, I will create an anonymized data set of usage info so that users can hack on extending the site to have stuff like comments, ratings, etc.

Owner

wbond commented Mar 11, 2013

I have it implemented locally, but now I am working on the channel server code so I can test it all. Hopefully I'll have something pushed up in the next few days.

I want to open source the website code, and share code between PC and the channel server, so now I am getting up to speed on Python 3 web development. Once the site is open-sourced, I will create an anonymized data set of usage info so that users can hack on extending the site to have stuff like comments, ratings, etc.

@schlamar

This comment has been minimized.

Show comment
Hide comment
@schlamar

schlamar Mar 11, 2013

Contributor

Ah great. FYI, I will help reviewing packages as soon as the v2 channel is ready.

Contributor

schlamar commented Mar 11, 2013

Ah great. FYI, I will help reviewing packages as soon as the v2 channel is ready.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Mar 22, 2013

Owner

Work at f85fd18 and 2c211dd.

I haven't really tested the new JSON structures with the new code yet. Still working on the new channel server, and there will be some further changes to the JSON, but I think just in adding some more useful fields.

Owner

wbond commented Mar 22, 2013

Work at f85fd18 and 2c211dd.

I haven't really tested the new JSON structures with the new code yet. Still working on the new channel server, and there will be some further changes to the JSON, but I think just in adding some more useful fields.

@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Mar 22, 2013

Contributor

Some thoughts after quickly reading through some parts of the commits:

  • // If the word "sublime" exists in the repository name, the name
    // is overridden by the "name" key.

  Does that mean if my repostory does not contain the "sublime" sequence the `"name"` field will have no effect? Seems weird.

- Does the `"includes"` key allow absolute URIs like "http://raw.github.com/fichtefoll/some_repo/some_repo.json"?

- > ```js
"sublime_text": "<3000"

This is probably irrelevant until you use a specific class for ST versions (I am planning on writing one but I'll be on vacation for the next 1 1/2 weeks). The plan is to make "~2", "2.x" or plain "2" match all ST2 versions. ST builds versions would be converted into like "2.0.1+2219.nightly" or "3.0.0-beta+3022.dev".
Anyway, some reference would probably be nice for these selectors, like I did here (which is btw. a bit different from your implementation).

  • 'description': info['description'] if info['description'] else 'No description provided'

  Better written as `'description': info['description'] or 'No description provided'`.

- For `semver_filter(versions)` try using the built-in `filter` function instead: `filter(SemVer.valid, versions)`.

- `_cmp_to_key(f)` works exactly like `functools.cmp_to_key(f)`.

Contributor

FichteFoll commented Mar 22, 2013

Some thoughts after quickly reading through some parts of the commits:

  • // If the word "sublime" exists in the repository name, the name
    // is overridden by the "name" key.

  Does that mean if my repostory does not contain the "sublime" sequence the `"name"` field will have no effect? Seems weird.

- Does the `"includes"` key allow absolute URIs like "http://raw.github.com/fichtefoll/some_repo/some_repo.json"?

- > ```js
"sublime_text": "<3000"

This is probably irrelevant until you use a specific class for ST versions (I am planning on writing one but I'll be on vacation for the next 1 1/2 weeks). The plan is to make "~2", "2.x" or plain "2" match all ST2 versions. ST builds versions would be converted into like "2.0.1+2219.nightly" or "3.0.0-beta+3022.dev".
Anyway, some reference would probably be nice for these selectors, like I did here (which is btw. a bit different from your implementation).

  • 'description': info['description'] if info['description'] else 'No description provided'

  Better written as `'description': info['description'] or 'No description provided'`.

- For `semver_filter(versions)` try using the built-in `filter` function instead: `filter(SemVer.valid, versions)`.

- `_cmp_to_key(f)` works exactly like `functools.cmp_to_key(f)`.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Mar 22, 2013

Owner

@FichteFoll Thanks for the taking the time to look over stuff.

  1. Yeah, I had noticed that ambiguity shortly after, and addressed it in c4b2e83
  2. Technically it should, yes.
  3. Yeah, in the interest of trying to keep things simple I think that just using build numbers for sublime_text is the easiest to understand. I'm not sure if there is real value in mapping them to semantic versions since you can't get that info from the sublime API.
  4. Good suggestions for the default description
  5. I can't use the built-in because I have to normalize date-based versions and 4+ dotted version (1.6.9.0) into semantic versions.
  6. That code has changed a bit since that original commit: https://github.com/wbond/sublime_package_control/blob/python3/package_control/versions.py
Owner

wbond commented Mar 22, 2013

@FichteFoll Thanks for the taking the time to look over stuff.

  1. Yeah, I had noticed that ambiguity shortly after, and addressed it in c4b2e83
  2. Technically it should, yes.
  3. Yeah, in the interest of trying to keep things simple I think that just using build numbers for sublime_text is the easiest to understand. I'm not sure if there is real value in mapping them to semantic versions since you can't get that info from the sublime API.
  4. Good suggestions for the default description
  5. I can't use the built-in because I have to normalize date-based versions and 4+ dotted version (1.6.9.0) into semantic versions.
  6. That code has changed a bit since that original commit: https://github.com/wbond/sublime_package_control/blob/python3/package_control/versions.py
@FichteFoll

This comment has been minimized.

Show comment
Hide comment
@FichteFoll

FichteFoll Mar 22, 2013

Contributor

@wbond

2# Better mention it in that case. ;)

3# I liked the "~2" selector thing really much, that's why I thought about it in the first place. However, it might seem a bit overkill and using build numbers (as I proposed in the first place) works well enough in conjunction with AND and OR selectors. (FYI, I was planning on writing a lookup table/dict where I could compare the current build against version increases and then prepend the respective stuff. No idea about the "nightly" and "dev" build segments though, I can only think of a lookup for each version which sucks.

5# You can do that anyway, just before passing it to the function. The current code does exactly what filter does.

Edit: By the way, instead of writing the semver_compat you could extend SemVer and modify the ver parameter before calling super(PacVer, self).__init__(ver, clean) in your __init__. That's what I was aiming for when I had the STVer thing in my mind. Or you just call semver_compat from within your extended __init__. (And don't forget to do the same with valid.)

class PacVer(SemVer):
    def __init__(self, ver, clean=False):
        ver = semver_compat(ver)
        super(PacVer, self).__init__(ver, clean)

    @classmethod
    def valid(cls, ver):
        ver = semver_compat(ver)
        return SemVer.valid(ver)

Edit2: Which then could result in filter(PacVer.valid, versions).

Ok, I edited this a few times because I'm kindy hurrying. No reply from me for the next ~3 hours.

Contributor

FichteFoll commented Mar 22, 2013

@wbond

2# Better mention it in that case. ;)

3# I liked the "~2" selector thing really much, that's why I thought about it in the first place. However, it might seem a bit overkill and using build numbers (as I proposed in the first place) works well enough in conjunction with AND and OR selectors. (FYI, I was planning on writing a lookup table/dict where I could compare the current build against version increases and then prepend the respective stuff. No idea about the "nightly" and "dev" build segments though, I can only think of a lookup for each version which sucks.

5# You can do that anyway, just before passing it to the function. The current code does exactly what filter does.

Edit: By the way, instead of writing the semver_compat you could extend SemVer and modify the ver parameter before calling super(PacVer, self).__init__(ver, clean) in your __init__. That's what I was aiming for when I had the STVer thing in my mind. Or you just call semver_compat from within your extended __init__. (And don't forget to do the same with valid.)

class PacVer(SemVer):
    def __init__(self, ver, clean=False):
        ver = semver_compat(ver)
        super(PacVer, self).__init__(ver, clean)

    @classmethod
    def valid(cls, ver):
        ver = semver_compat(ver)
        return SemVer.valid(ver)

Edit2: Which then could result in filter(PacVer.valid, versions).

Ok, I edited this a few times because I'm kindy hurrying. No reply from me for the next ~3 hours.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Mar 22, 2013

Owner

I will document 2#.

5# Right, so I can either write a function that I pass to filter, or a write a function that wraps filter. Either way, I want that logic in versions.py.

Owner

wbond commented Mar 22, 2013

I will document 2#.

5# Right, so I can either write a function that I pass to filter, or a write a function that wraps filter. Either way, I want that logic in versions.py.

@wbond

This comment has been minimized.

Show comment
Hide comment
@wbond

wbond Aug 14, 2013

Owner

This 2.0.0 release added support for this.

Owner

wbond commented Aug 14, 2013

This 2.0.0 release added support for this.

@wbond wbond closed this Aug 14, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment