Some .for files are text.

[larsbrinkhoff]

As of late, many .for files have been created which are neither FORTRAN nor Forth:
https://github.com/search?q=extension%3Afor+weekly&type=Code

The contents of these files are identical. I added it as a Textsample.

[pchaigno]

There isn't currently a heuristic rule to distinguish Text from FORTRAN and Forth. Because of #2042 if we add .for for Text the heuristic for FORTRAN and Forth won't be used anymore. It seems risky for only 107 files :/

[larsbrinkhoff]

@pchaigno Sorry, I added Text to the heuristic rule for .for, but lost it before I made this pull request. So good catch.

I'll update. Now, the heuristic rule will only return a result for fairly obvious matches of Forth or FORTRAN. For ambiguous cases, it will fall back to the bayesian classifier. And the Text sample will catch the "weekly" file.

Do you still object?

[pchaigno]

I'm not objecting, I just think we should be careful :-)
And we probably should fix (or at least clarify) the situation with #2042 before. Currently, if you add Text to the Forth and FORTRAN heuristic, .F files may be recognized as Text :/

[larsbrinkhoff]

.F files may be recognized as Text :/

I don't think that can happen (at this moment), because the heuristics never return Text.

But you are right that we should be careful. Unless this PR is shot down, I'll request 1000 samples and check the results before and after.

[pchaigno]

I don't think that can happen (at this moment), because the heuristics never return Text.

Oh, you're right; I misread you :/
If it works with the classifier, perfect! You'll have to test that the new Text sample doesn't break other Text detections though.

[larsbrinkhoff]

I'm all for checking thouroughly! :-) Just awaiting the words of the elders.

Also, these files seem to come in at a rate of a few a week, and they cause the entire repo to be misclassified as Forth. They are starting to overwhelm true Forth repositories.

[bkeepers]

I'm curious what these .for files actually are. It looks like some data format. I wonder if we should be adding another definition for it.

[larsbrinkhoff]

Me too. I believe the original file came from here: http://www.cpc.ncep.noaa.gov/data/indices/

I emailed the contact at that page, but haven't got any reply yet.

There are a few other .for files there. As far as I can see, they are columnar data separated by whitespace, and possibly including title and column headers. My best guess is that it's "formatted data".

[larsbrinkhoff]

Tested with 1000 Text samples: no change in classification.

Tested with 1000 .forsamples:

	Before	After
FORTRAN	577	577
Text	0	415
Forth	423	8

I checked that those Forth files were indeed misidentified.

[larsbrinkhoff]

Received word from the original author:

".for" is short for "formatted".

[larsbrinkhoff]

According to http://www.weather.gov/disclaimer, the sample file is in the public domain,

as long as you do not: 1) claim it is your own (e.g., by claiming copyright for NWS information -- see below), 2) use it in a manner that implies an endorsement or affiliation with NOAA/NWS, or 3) modify its content and then present it as official government material. You also cannot present information of your own in a way that makes it appear to be official government information.

[larsbrinkhoff]

Another new repository with plenty of .for files:
https://github.com/OpenSceneryX/Library/search?q=extension%3Afor

Before my proposed change, it's classified as 48% Forth and 33% Python. After, 62% Python and no Forth.

[larsbrinkhoff]

Ping @bkeepers @arfon. Any thoughts?

[arfon]

Ping @bkeepers @arfon. Any thoughts?

I think we should probably merge this as it's a growing issue (178 files now). I'd really like to get #2042 resolved soon too.

[larsbrinkhoff]

The Travis CI build is erroring now, because of

apt-get --assume-yes download libicu48 libicu-dev
Err Downloading libicu48 4.8.1.1-3ubuntu0.1
404 Not Found [IP: 91.189.91.15 80]
Err Downloading libicu-dev 4.8.1.1-3ubuntu0.1
404 Not Found [IP: 91.189.91.15 80]
ar p vendor/apt/.deb data.tar.gz
tar -vzxC vendor/debs --strip-components=2
ar: vendor/apt/.deb: No such file or directory

I'm not sure there's anything I can do about that?

[arfon]

The Travis CI build is erroring now, because of

@aroben - I had a look at fixing this earlier and didn't find an obvious solution. It looks like we need an apt-get update before the install but we can't do the recommended as we can't use sudo on container builds.

Perhaps we should add the libicu48 deb to a local directory and install from there?

[bkeepers]

I can't wait until we come up with a way to return "unknown" for files that don't have a high score from the classifier.

I don't love that we are classifying this as Text. Should we add a Formatted entry? Is there a spec for the file format anywhere?

[larsbrinkhoff]

I can't wait until we come up with a way to return "unknown"

Me too! Oh oh, and also a way to search for files and repos without an assigned language.

I don't love that we are classifying this as Text. Should we add a Formatted entry? Is there a spec for the file format anywhere?

Agreed. From looking at various .for files, I'm quite sure there is no spec. Or maybe the spec would be "numbers formatted into whatever format looks pretty, plus some text that looks nice".

But anyway, I was thinking maybe something like this:

Formatted Data:
  type: data
  group: Text # maybe?
  extensions:
  - .for
  # maybe .f too, sigh

[larsbrinkhoff]

@arfon @bkeepers Added a commit to introduce a new Formatted language. Does this look ok?

[arfon]

This looks good to me. @bkeepers do you think we should add group: Text?

[bkeepers]

Looks good to me 👍

do you think we should add group: Text?

No, it doesn't seem like this should be text to me. It's data, right?

[arfon]

It's data, right?

Data encoded in plain text :-\

[arfon]

Seriously though, I don't care that much. I could be convinced either way.

[larsbrinkhoff]

I merged the two commits into one. Leaving out group: Text.

[arfon]

@larsbrinkhoff - probably won't cut a new gem until early next week.

[larsbrinkhoff]

That's fine. Thanks!