Added option to allow the user to make the contributions listing dependent on the PDE version

[joelmoniz]

This fixes #2581.

This would require a new field in the contributions.txt and the .properties files. The field would consist of a comma separated list of integers, separated by dashes (-) for ranges. If the last entity has the right side of its - without a number, it implies that the contribution is compatible with all revisions of Processing after that number (including the number itself). For an example, refer the documentation here.

[benfry]

Does this allow people to post multiple versions of what is technically the "same" contribution, but compiled for different versions of the PDE? (i.e. Python Mode and others will need to post two versions, one for 3.x, another for 2.x)

[joelmoniz]

Oops. No it doesn't. All it does at present is show whether or not a contribution is compatible with the present Processing version.

[shiffman]

What is the scenario we are looking for? Is it. . .

The library maker posts two versions of the same library at different urls, i.e.

shiffman.net/libraries/box2d_2/box2d_processing.zip
shiffman.net/libraries/box2d_2/box2d_processing.txt

shiffman.net/libraries/box2d_3/box2d_processing.zip
shiffman.net/libraries/box2d_3/box2d_processing.txt

and then list the compatible versions in the properties file? Both listings then go in to contributions.txt?

And then the Processing user has separate sketchbooks for 2 and 3?

Just trying to understand this issue a little better, apologies if I'm behind.

[benfry]

I guess? I don't have an exact solution, but if we're going to start breaking things in 3.x, we can't simply force the users and contributors over from 2.x and say "abandon your old libraries, modes, and tools and use the 3.x pre-releases" since we're in the alpha/beta stage (and will be for a few months).

A concrete example: CMU would like to use Python Mode for Fall classes. 3.0 final will likely not be ready for Fall classes, so they need to use 2.2.1. We'd like the Python Mode folks to support the in-progress changes in 3.x as we go (for as many users as possible) as well as 2.x (for those courses).

[shiffman]

Another option would be to have a single library have multiple compiled jars, I think this maybe a bit too cute / unnecessarily complex.

box2d_processing/library/2/box2d_processing.jar
box2d_processing/library/3/box2d_processing.jar

cc @prisonerjohn

[shiffman]

I realized I'm being very silly b/c 2.0 won't know about this folder structure. So I think we have to do multiple contribution listings / sketchbooks. .

[joelmoniz]

It might be possible to have a single contribution.

For example, consider the CoffeeScript Mode. How the existing jars are loaded is by searching for the folder in the CoffeeScript Mode's folder that exactly matches the name of the contribution's type, i.e. in this case it searches for the mode folder, and loads the jars of this folder and this folder alone. So what about having all the compatible contributions listed in the Contributions Manager only once (with a single .properties file, and a single folder), but having contributions compatible with only the 3.x versions in a separate folder? The new folders would be something like mode2, mode3, mode4 etc., each with a text file in them mentioning for which all revisions the mode is compatible. The original mode folder will be assumed to be compatible with all revisions present in the properties file, but not present in any of the other jars' folders.

Is this method alright? Any potential pitfalls? Any way of simplifying it?

[federicobond]

I second @joelmoniz's idea. We just have to make sure that once contributions stop supporting 2.x we are left with the structure that we want and no cruft from previous versions is needed.

[benfry]

@joelmoniz This wouldn't work with the video library, unfortunately, which is around 100-120 MB, and would become 200-250 MB in order to support both 2.x and 3.x.

[joelmoniz]

Then what about adding one more field to the contributions.txt, say download3, and having newer versions use that link? This download3 would be a list of (link,compatibleVersion) pairs, maybe, and the contribution could be installed and extracted to a mode2, mode3, mode4, etc. folder within the contribution's folder if the contribution is already installed?

cc @fjenett

[joelmoniz]

@benfry

Shall I begin implementation as follows?:

1. Add a new field that contains a link-version list, as follows:

mode
category=
authorList=[Florian Jenett](http://bezier.de/)
name=CoffeeScript Mode
sentence=Adds a CoffeeScript Mode based on Processing.js
url=https://github.com/fjenett/coffeescript-mode-processing
prettyVersion=0.0.7
paragraph=CoffeeScript Mode is based on JavaScript Mode and runs on top of Processing.js. It runs through a server and inherits the template system from JavaScript mode. CoffeeScript is a functional language that compiles to JavaScript and was created by (Jeremy Ashkenas)[https://github.com/jashkenas/coffee-script].
version=267
download=http://bezier.de/processing/modes/CoffeeScriptMode.zip
**compatibleVersions=200-230
linkVersionList=<225-228,http://bezier.de/processing/modes/CoffeeScriptMode2.zip>,<229-,http://bezier.de/processing/modes/CoffeeScriptMode3.zip>**
id=070

The contributions manager will automatically use the appropriate link to download the contribution.

So, above, the CoffeeScriptMode.zip will be used for all revisions not present in the list, CoffeeScript2.zip for revisions 225-228,CoffeeScript3.zip for revisions 229 onwards.

2. When installing the mode, if a CoffeeScript folder already exists, simply install something like a ```mode_225-228``` folder for versions 225-228, and a ```mode_229on``` for modes 229 onwards, with common examples and .properties files.

Have I missed anything or left something unexplained? This should not affect the present revisions in any way as far as I can tell, but should enable the newer revisions to use contributions that are compatible with them without the confusion that would be caused (especially to newer users) by including the same contribution multiple times.

Thanks.

[fjenett]

I wonder if now that we see to move away from simple properties listings we should switch to a more powerful syntax? JSON maybe? ... otherwise we will need to write complicated parsing code and people will have to learn our syntax.

I think we could get away with a versions listing:

versions : [
    <220 : http://xxx.zz/aa/bb/cc/MyLib01.zip,
    221-255 : http://xxx.zz/aa/bb/cc/MyLib02.zip
]

[prisonerjohn]

We've been chatting by email about the contributions.txt file and here's what we're thinking will work well and be simple enough to implement:

• Keep the old contributions.txt around for a little bit while we're still supporting Processing 2.x.
• Create a new PContributions.txt file which we'll use for Processing 3.0 onward.
• Update the build script so that it generates both contributions.txt and PContributions.txt. Any entry with a "compatibleVersions" field can go in the new list, and any entry missing that field or with a value of "2.x" can go in the old list.

I like @fjenett's idea of switching to a more powerful syntax like JSON, I think it will simplify everything down the line. However, that will require that the parsing engine for the Contribution Manager and the site be re-written.

I feel like the original idea of having two versions of the library at different URLs with two listings in the contributions file is the way to go. Keeping everything separate is clean and will save us headaches. Besides, if the JARs are different, I'm assuming that implies that the sources and reference will also be different.

And the way I understand it, entries will only be duplicated if there are different versions of the contribution. So, if for example, my library has "compatibleVersions=220-300", that will still be a single entry, and a single directory in the libraries folder for both Processing 2.x and 3.x

[joelmoniz]

So the contributions manager will have to parse and display both
contributions. txt as well as PContributions.txt, right?

And would the 3.0 contributions be stored in a separate folder? Also, when
loading the contribution, a check would have to be performed to ensure
contribution compatibility, right?

Thanks.

[prisonerjohn]

No, the contributions.txt is just sticking around for Processing 2.0 support, so the new contribution manager in Processing 3.0 onwards only needs to consider and parse PContributions.txt. Does that make sense?

As for storing the contributions, I'm leaning towards putting everything in the sketchbook directory as we're doing now, but maybe I'm missing something... I'm assuming most libraries will be compatible with Processing 2.0 and 3.0, so will not require separate versions.

[REAS]

Processing 3 will create a new "sketchbook." When people download Processing 3, we have a plan to ask people if they would like to move over their sketches. We don't plan to attempt to move over Libraries, Modes, and Tools. So people will do a fresh install of libraries etc. from the contribution managers within the 3.0 software. (These managers will only list the 3.0 compatible stuff.)

[joelmoniz]

Where should the 3.x contributions that have the same name as the 2.x contributions be stored in case a duplication (2 separate versions, I mean) of the contribution is required?

EDIT: To clarify, how exactly will contributions that require 2 versions be stored in case both versions are to be installed? Will it be ensured that the names of the contributions that require 2 different versions will be different, so that 2 folders with the same name are not required?

[fjenett]

Trying to wrap this into a plan, sorry if this duplicates the info above ...

We continue to support contributions.txt and the old sketchbook system for Processing 2.x and below.

New "PContributions.txt" and new sketchbook folder for Processing 3.0

PContributions.txt will now hold an compatibility listing, giving information about which version of a contribution is compatible with what version of Processing (per release number).

The new sketchbook needs a system to separate and distinctably store versions of the contributions. I propose:

libraries/
   MyLib/
      1.x-alpha/
         library/
         examples/
         ...
      2.4-rc/
         library/
         examples/
         ...
   AnotherLibrary3D/
      1.2/
  ...

And PContributions.txt entry for MyLib would then have:

versions : [
    <220 : [ 1.x-alpha, http://xxx.zz/aa/bb/cc/MyLib01.zip ],
    221-255 : [ 2.4-rc, http://xxx.zz/aa/bb/cc/MyLib02.zip ]
]

... listing not only URLs but also the version behind that URL. This allows us to check wether a contribution is already installed and only download if needed.

... i think "PContributions.txt" should be named contributions.json as it is not a class and there is no need to make it look like one. The new ending will separate it from the old one.

Anything missing?

[prisonerjohn]

I can't really speak on the sketchbook folder structure, but I'm down for the new contributions file being in JSON format and being named contributions.json. @alignedleft @REAS this would imply changes to the way the libraries, tools, and modes pages are generated on the site.

I could whip out an updated build script today or tomorrow if we go this route.

[benfry]

This has grown to be more complicated than necessary... Because we're splitting 2.x and 3.x sketchbooks, let's not get into the multiple versions of a library installed at a time. For the 'versions' field, I think a minimum and maximum version (both optional) should be sufficient. If we really need interrupted ranges, there's a more serious problem, and it shouldn't be solved this way.

In the end, we want people to install the latest 3.x (once it's ready), but support 2.x for the meantime (since it's not). We can't tie ourselves in knots trying to solve every scenario that someone might get themselves into, and wind up with fragile code that's trying to do too much.

Fine with moving to a .json format if needed. But keep in mind that that as it stands now, it's a couple of lines to parse the file, and it's not broken as far as I know. I think the PContributions.txt name was a joke, contribs.txt should be fine if we stick with the text format (or contributions.json if we go to json).

[REAS]

Is it possible to move to JSON without requiring people with existing libraries in good working order to write a new contribution data file? Letting contributors keep their libraries "as is" for the 3.0 launch if they are in good working order is the largest priority.

[benfry]

@REAS That's a good point... Let's not annoy Library/Mode/Tool authors with a needless change. I think in that case, let's stick with the old format, unless there's a strong reason to change it. If we get rid of arrays of revision ranges, we shouldn't need JSON.

[prisonerjohn]

So, should we add two fields to the current properties, something like minTestedVersion and maxTestedVersion?

If either is missing, they would default to "221", and that would require no immediate changes to existing libs.

[aengelke]

@REAS As the contribution file gets parsed in the generation, a change to JSON would work without a change to the libraries.

[prisonerjohn]

@aengelke This is true, but if we're asking contributors to provide multiple versions of their libs for different versions of Processing, we'd probably also need to change the submission process as well as the .properties file format.

[benfry]

@prisonerjohn Yeah, though I think we should leave 'tested' out, since I don't think we want to mix whether things are tested or not (which would be a warning) with a hard limit on what's compatible (which would lead to the contribution being hidden from the listing).

[joelmoniz]

Here's what I understood as to how to proceed. Is this correct? Have I missed something out?:

• Have 2 sets of contributions.txt, one for 3.x, and another for 2.x (although each version won't really know that the other exists). Possibly keep different link for the two?
• Have a minVersion and maxVersion. If the maxVersion is less than the current version, ignore contribution. However, some contributions will have 2 sets of contributions, one for the 2.x, and another for the 3.x. In this case, install the 3.x one, provided the 3.x version lies between the minVersion and maxVersion.
• The 2 sketchbooks (of 2.x and 3.x) will be independent of each other, each having their own sets of contributions.
• Like above, compatible contributions will be shown via a checkbox.

Is this alright?

EDIT: Really, really sorry if you got multiple emails/notifications. There was something wrong with my browser, and the updated message kept on not showing, which resulted in things being posted multiple times when I kept trying again...

[benfry]

I think that all sounds correct, though from the standpoint of what actually changes:

• The current 2.x contributions.txt file continues to live at the same URL, and will remain untouched
• We add a new 3.x contribs.txt file at a new URL, seeded with the contents of the current contributions.txt file.
• The manager code in 3.0a3 and later knows to check min/max version found in contribs.txt, and dim out/disable installation of incompatible contribs. (Optionally, the dimmed out/incompatible contribs can be hidden with the checkbox. But we should default to showing all available, and disallow installation of a contrib that's known to be incompatible, regardless of the checkbox setting).
• As authors make changes for 3.x, they will be added/updated in the contribs.txt file. If supporting a 2.x and 3.x version of the same library, they will be considered a "different" library, because their sources will be from two different URLs.
• If a library breaks in 3.x, we remove it from contribs.txt.

Also realizing that it should be named min/max revision, not version. The revision number for 2.2.1, for instance, is 0227 (it should be listed in the file as 227, since 0227 is technically "octal" encoding).

For documentation, we should tell authors to only use maxRevision (or minRevision) when it's known to break in a later (or earlier) release, and not treat it as a "last known revision to work" unless they want to update their library with each Processing release.

[prisonerjohn]

I updated some bits and pieces to get this going.

• The build script now accepts minRevision and maxRevision fields, set to 0 by default. The output looks like this.

• The build script can also do its own filtering using these fields, and can generate the 2.x contributions.txt as well. Instructions to do so are here.

• The library template was updated to include these fields.

• This is the documentation I'm adding. (Is it clear enough?)

  # The min and max revision of Processing compatible with your library. 
  # Note that these fields use the revision and not the version of Processing, 
  # parsable as an int. For example, the revision number for 2.2.1 is 227. 
  # You can find the revision numbers in the change log: https://raw.githubusercontent.com/processing/processing/master/build/shared/revisions.txt
  # Only use maxRevision (or minRevision), when your library is known to 
  # break in a later (or earlier) release. Otherwise, use the default value 0.
  

Let me know if this works.

[joelmoniz]

Thanks!

@prisonerjohn : To clarify,

• A value of 0 in both minRevision and maxRevision would imply that it is compatible with all revisions.
• A value of 0 in minRevision and x in maxRevision would imply that it is compatible with all revisions before x.
• A value of x in minRevision and 0 in maxRevision would imply that it is compatible with all revisions after x.
• A value of x in minRevision and y in maxRevision would imply that it is compatible with all revisions between x and y (including them).

Could you please let me know if this is correct? Thanks again.

[prisonerjohn]

Yep, thats it!

On Wednesday, August 13, 2014, Joel Moniz notifications@github.com wrote:

Thanks!

@prisonerjohn https://github.com/prisonerjohn : To clarify,

• A value of 0 in both minRevision and maxRevision would imply that it
  is compatible with all revisions.
• A value of 0 in minRevision and x in maxRevision would imply that it
  is compatible with all revisions before x.
• A value of x in minRevision and 0 in maxRevision would imply that it
  is compatible with all revisions after x.
• A value of x in minRevision and y in maxRevision would imply that it
  is compatible with all revisions between x and y (including them).

Could you please let me know if this is correct? Thanks again.

—
Reply to this email directly or view it on GitHub
#2746 (comment)
.

Elie Zananiri
@prisonerjohn

[joelmoniz]

Thanks!

I'm done with the incompatibility aspect of things. I've replaced the "compatibleVerisons" field with the "minRevision" and "maxRevision" fields. Incompatible contributions are now dimmed out, and a checkbox to hide them is also present. How would I go about proceeding further?

[benfry]

Is this ready for me to merge in its current state?

[joelmoniz]

I think the other 2 merges caused a merge conflict. I'll resolve them in a jiffy.

[joelmoniz]

Alright. Everything should be auto-mergeable now, and everything ought to work as discussed.

[benfry]

Thanks, here we go!

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.