Enhancement: Perform the Great Fork Merge

[lcn2]

The Great Fork Merge

The Great Fork Merge will occur when the multi-thousand commits that the temp-test-ioccc repo is ahead of the Official IOCCC winner repo are bright back to main Official www.ioccc.org web site.

TODOs

In order to perform the Great Fork Merge, the following tasks need to be completed:

• Revise status.json format.

• Write a new tool to generate the new top level status.html from status.json

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file.

• Fix name www under RHEL 9

Port the bin tools tools to run under RHEL 9 version of Linux. With this port, the IOCCC judges should be able to use these tools on a wide enough variety of systems for their purposes.

• Complete issue Enhancement: Address XXX lines in top level files #2008

• Complete issue Enhancement: improve the consistency of winning directories #3

• Verify that the format of markdown files is good

See also comment 1993767060.

• Complete issue Enhancement: add templates for issue forms #858

• Complete issue Enhancement: Improve the IOCCC manifest #1933

• Move all FAQ content into their own menu section

See comment 9192931.

• Verify that the top level README.md contains suitable content for a GitHub repo README file.

The exception, of course, is the initial text down to the XXX that indicates this is a test repo.

• Complete issue Enhancement: improve markdown links to FAQ entries #2417

• Complete issue Enhancement: improve the look and feel of the constructed HTML files #2006

Improve the look and feel of the constructed HTML files.

• Complete issue Enhancement: Reorganize the FAQ #2682

Reorganize the FAQ.

• Complete issue #979 in the mkiocccentry repo

• Complete issue #955 in the mkiocccentry repo

• Complete issue #956 in the mkiocccentry repo

Prepare the mkiocccentry repo for a code freeze and release.

• Reformat JSON indented with multiples of 4 ASCII spaces and no TABs

See also GH-issue 931 from that other repo.

NOTE: JSONPath.sh with only -S -A and without -T may be helpful with this coversion.

NOTE: The bin/jprint-wrapper.sh tool should also be updated.

• Complete GH-issues/931 from the mkiocccentry repo

This will be in keeping with the completed TODO the JSON indented with multiples of 4 ASCII spaces and no TABs.

• Review and improve bin/README.md

Now that man pages for bin/ tools (issue #2009) are not planned, we need to review and likely improve the bin/README.md documentation.

We need to add some notes about these recent additions:

• bin/all-jfmt.sh
• bin/combine_author_handle.sh
• bin/jval-wrapper.sh
• bin/manifest.csv.entry.awk
• bin/manifest.entry.json.awk

• Create a new issue for recovery of very old entries in news.md

Issue #2686 was created to satisfy this TODO item.

• Build bin tools making manifest.numbers no longer necessary and make .entry.json the primary "entry data truth"

See comment-2189364148.

• Add an FAQ about how to handle de-obfuscated code

See comment 919283.

See comment-2198635072.

This TODO was changed from referring to a "spoiler" into a "de-obfucated" educational emphasis. In that regard we should NOT use terms like "spoiler" in the FAQ as well as NOT using "spoiler" in the entry's README.md file.

At a most, README.md should suggest that the reader might wish to study the prog.c code first before going on to review the de-obfuscated "alt" code.

See also comment-2198807715.

After completing the FAQ entry how to handle de-obfuscated code, consider if needed, revising entries with existing de-obfuscated code AND if needed, fixing cases where the entry README.md makes reference to something being a spoiler.

• While OT, create an issue for a that performs JSON semantical checks on the .entry.json files in mkioccentey repo

See comment-2172105426 and see comment-2174364721.

• While OT, fork the jparse/ directory from mkioccentey repo tree into @xexyl's jparse tree

See comment-2197371351.

• Resolve issue [Bug] bin tools are using jstrencode(1) when they should be decoding JSON encoded strings #2752

• Create an initial text for details of the registration process as a new FAQ 1.4

Add a "see also" link to the new FAQ 1.4 in FAQ 0.0 subsection 2.

If the registration process is not "screenshot ready" by the time this TODO is addressed, put in a "stay tuned for images" placeholder into this new FAQ.

• Create an initial text for details of the submit process as a new FAQ 1.5

Add a "see also" link to the new FAQ 1.4 in FAQ 0.0 subsection 5.

If the ** submit process** is not "screenshot ready" by the time this TODO is addressed, put in a "stay tuned for images" placeholder into this new FAQ.

• Carefully reread and review the next rules and guidelines

• Carefully reread and review the FAQ - Especially content related to how to enter the IOCCC

In addition to the useful check for typos, wording, and broken links: review for suitability and readability for gong live with on the main web site after the Great Fork Merge. Update faq.md if and as needed.

MOST IMPORTANT / HIGHEST PRIORITY: The most important is to review for suitability and readability the "FAQ 0.0" / "How to enter" stuff and the related pages (such as "next/*.md"), as well as the rules and guidelines. This is what people who have been waiting to enter the IOCCC will use most often.

• Cleanup from 2024 testing

We need to cleanup from testing tools, so we need to remove:

• 2024 from .gitignore
• 2024 from .top
• 2024 from Makefile

These may have added as part of tool testing and need to be removed before we proceed along the Great Fork Merge process below.

• Inspect and sanity check the remaining TODO items

Add, modify and/or remove things from the remaining TODOs if/as needed.

• Determine that we are ready to perform the Great Fork Merge

• Announce a code freeze

Add a "GFM-code-freeze" (Great Fork Merge code freeze) git tag to top commit of the master branch.

In a comment in this issue #2239, announce that: General pull requests for the temp-test-ioccc repo will not be accepted as are beginning work on the Near Final TODOs.

Getting ready for the Near Final TODOs

NOTE: Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order (preferably in the same calendar day) before the Great Fork Merge takes place.

• Remove all trailing whitespace from markdown files

Use this to look for any whitespace from markdown files

find . -name '*.md' -print0 | xargs -0 grep -E -l '[[:space:]][[:space:]]*$'

• Verify the markdown code blocks are properly indented

Use:

find . -name '*.md' ! -name markdown.md -print0 | xargs -0 pcregrep -M '^``.*\n[^|`\n \t]'

• Verify that the Nu Html Checker finds no errors, warnings nor info messages for all generated HTML

After making sure that the temp-test-ioccc repo is up to date and related GitHub pages have been rendered, use the ✓ on the navbar to check all generated HTML pages.

Fix any errors, warnings and info messages reported. Update the temp-test-ioccc repo and recheck those pages.

• Update FAQ 1.2 with there date of Great Fork Merge

In FAQ 1.2, look for the <!-- XXX - Fill in the date when Great Fork Merge happens --> and update the date as for that section as needed.

• Change bin tools to use www.ioccc.org

Change default URLs and REPOs to refer to the https://www.ioccc.org web site and winner repo.

• Change references to use www.ioccc.org instead of temp-test-ioccc

Change references as needed given their context.

This includes markdown files, HTML files, var.mk, Makefiles, text files, etc.

Try:

make www
find . \( -name .git -o -name NOTES \) -prune -o -type f -print0 | xargs -0 grep -l -F temp-test-ioccc

Also change the comment in var.mk

• In mkiocccentry repo, Change references to use www.ioccc.org instead of temp-test-ioccc

See GH-#issuecomment-2403175498.

• Re-release the mkiocccentry repo

• Rebuild web tree via make www

After re-installing (both from the top level and the jparse sub-directory) from the mkiocccentry repo, perform a make www and see if anything changed (there shouldn't be any changes).

• Scan repo to verify that references to temp-test-ioccc have been processed.

The only exceptions should be historic references found in news.md, news.hml, faq.md and faq.html.

• Add news event to news.md announcing the new www.ioccc.org web site.

Edit news.md as needed.

Perform make www and commit any changes.

Near Final TODOs

NOTE: Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order (preferably in the same calendar day) before the Great Fork Merge takes place.

• Verify that the installed location(1) tool is up to date

Be sure the mkiocccentry repo is up to date and perform a make install in that typo.

• REJECTED: Purge repo of previous *.tar.bz2 files and reduce Git repository size

We have rejected the action because we see LITTLE VALUE and TOO MUCH RISH. So we have rejected these 2 TODO steps.

See GH-issuecomment-2430718370 for a summary of the test process.

We will perform the following actions:

git mv 2011/dlowe/dlowe-aux-data.tar.bz2 HOLD
git commit -m'move dlowe-aux-data.tar.bz2 to HOLD'

git remote -v
git filter-repo --path-glob '????/*.tar.bz2' --invert-paths --force
git remote add origin git@github.com:lcn2/trim-temp-test-ioccc.git

git mv HOLD 2011/dlowe/dlowe-aux-data.tar.bz2
git commit -m'move HOLD back to dlowe-aux-data.tar.bz2'

for year in [12][0-9][0-9][0-9]; do
  touch "$year/$year.tar.bz2"; git add "$year/$year.tar.bz2";
done

for year in [12][0-9][0-9][0-9]; do
  for dir in $(< "$year/.year"); do
    entry_id=${dir/\//_};
    touch "$dir/$entry_id.tar.bz2";
    git add "$dir/$entry_id.tar.bz2";
  done;
done

git commit -m'add empty tar.bz2 files until we do make update'

git repack
git prune-packed
git reflog expire --expire=now --all
git gc --aggressive --prune=now
git fsck --full

git push --all --force
git push --tags --force
git push --set-upstream origin master

FYI: See:

• GitHub notes on Removing sensitive data from a repository
• comment-2159159871
• stack overflow replies
• How to clean up the git repo and reduce its disk size
• GH-issuecomment-2430718370

NOTE: The make update in a TODO farther below will add back the needed *.tar.bz2 files.

NOTE: The make gen_years will generate Warnings of the form: "bin/gen-years.sh: Warning: compressed tarball for YYYY: yyyy is empty: yyyy/yyyy.tar.bz2".

This is OK and will be corrected once make update or make tar is done.

• REJECTED: Perform make update on macOS and push out any changes

• Verify that running make update again on macOS changes only:

• sitemap.xml
• status.html
• status.json

If so, then revert those changes by:

git restore sitemap.xml status.html status.json

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

• update README.md and index.md

Remove the lines between:

<!-- XXX - This entire section goes away during the final stages of the Great Fork Merge -->

and:

<!-- XXX - remove down to here in the final stages of the Great Fork Merge -->

• Verify that the temp-test-ioccc repo is clean

Verify that:

git status

reports:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

• Verify that the temp-test-ioccc repo is up to date

Fetch any last minute changes:

cd docroot/temp-test-iocc

git fetch && git rebase

git clean -f

git status

The last command should report:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

• Perform make update on macOS

• Commit all changes and push the result to the temp-test-ioccc repo

git commit -m'final pre-Great Fork Merge'

git push

• Re-Verify that the temp-test-ioccc repo is up to date

Fetch any last minute changes:

cd docroot/temp-test-iocc

git fetch && git rebase

git clean -f

git status

The last command should report:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

• Re-Verify make update on macOS changes changes only:

• sitemap.xml
• status.html
• status.json

If so, then revert those changes by:

git restore sitemap.xml status.html status.json

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

FINAL REVIEW

• REJECTED: Pause and ask for a review of readiness

We expect this period to be between 2024 November 1 and 2024 November 15.

Please review the effect of all of the above completed to do. Please limit PRs to only critical problems, and broken links, and linked to the wrong file, and failure to the official website and repo, etc.

• REJECTED: Incorporate any PRs as a result of the previously mentioned review

Accept any such PRs, if needed modify the result after accepting, and finally do a make update.

Great Fork Merge

• Perform the Great Fork Merge

Click Create Pull Request for the Official IOCCC winner repo.

Inspect the pull request for the Official IOCCC winner repo.

Accept and complete the pull request.

Post Great Fork Merge TODOs

Once the Great Fork Merge occurs and the official IOCCC winner repo and related Official www.ioccc.org web site has been updated, these TODOs need to be performed on the official IOCCC winner repo:

• Copy issue Enhancement: recover old and historic news items and make them available via news.md #2686 into the official IOCCC winner repo

Close down issue #2686 as "won't fix" in this repo.

• Close issue Question: Issues that aren’t really major issues but are still issues #5

• Transfer any remaining issues to the official IOCCC winner repo

• Remove the website associated with the temp-test-ioccc repo

• Verify that the official IOCCC winner website links work

Fix any problems found on the Official www.ioccc.org web site by editing the official IOCCC winner repo as needed.

• Announce on Mastodon that the Great Fork Merge has happened and the official IOCCC web site has been updated

• Mark the temp-test-ioccc repo read-only

From temp-test-ioccc repo settings click on the ((Archive this repository)) button.

This will Freeze the temp-test-ioccc repo but leave it in place for historic purposes.

[xexyl]

Please assign this to me.

And as I noted before: if needed I would like to update my script that updates the status.json file.

Thanks.

I will have to look at this issue later on today as it seems rather important!

[xexyl]

Also I am not sure if I can finish the year README files before the 10th but I should be able to finish them sometime soon anyway.

But for now I must go do other things again. Back in a while.

[xexyl]

In order to perform the Great Fork Merge, the following tasks need to be completed:

• Revise status.json format.
• Write a new tool to generate the new top level status.html from status.json

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file.

It's funny you should say that .. the generating the file. Do you mean creating the file rather than modifying it? I actually had the thought of adding an option to the script to do that and I seem to recall that I almost did do it. It would be easy to do. But since the format is not necessarily the final format I will hold off on that.

• Complete issue Enhancement: improve the consistency of winning directories #3

The anticipated completion date for this issue is 2024 March 10 2:30 (2:30AM) Pacific.

Like I said over there mostly it is done: except for the YYYY/README.md files. But I can do those after 10 March if necessary so I have no problem with it being closed. You were going to take care of the silencing the warnings and right now would be fine as I don't anticipate modifying any Makefiles or if I'm honest for now anything at all (except maybe the FAQ which you asked me to look at). That probably will hold for a day or two at least but even if I do make other changes it'll likely only be the YYYY/README.md files. I say I might not finish it before 10 March but I might yet: I just have to have the energy and time and the former seems to be a problem lately I'm afraid. Even so I might be able to do some of it today: that remains to be seen however.

• Complete issue Enhancement: add templates for issue forms #858

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

• Complete issue Enhancement: Improve the IOCCC manifest #1933

Ah yes ... I forgot. I have to update descriptions and also check the html files! It's been very hectic and I totally forgot about this. I was also holding off as you were planning many changes that would conflict. Those changes are now done so I can look at this sometime soon too. If I don't please remind me.

• Complete issue Enhancement: improve the look and feel of the constructed index.html files #2006

Right ... well this one might be good to look at when I look at the manifest too. After all the result of the manifest is in the index.html files.

• Complete issue Enhancement: Address XXX lines in top level files #2008

Well some of this cannot be done until the submit server is ready, right?

Near Final TODOs

Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order before the Great Fork Merge takes place:

• Change HTML references to use www.ioccc.org

In the (few?) cases where a absolute URL is required, change that URL to be under https://www.ioccc.org and NOT this repo's top level URL.

sgit(1) should be useful for this one! :)

• Add news event announcing the new www.ioccc.org web site.

Edit news.md as needed.

That's a great idea .. looking forward to see what you have in mind! Almost eager enough to try and get myself to work on YYYY/README.md files today but I think that might be a mistake with how tired I feel :(

• Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

• Verify manifest.numbers agrees with the local temp-test-ioccc repo

How do you mean verify that it agrees with it when it's in the repo? Though again I guess this might have to be something you do so I don't need to worry about it: updating the manifest I do have to worry about but perhaps not this step?

Perform the 12 steps as noted in comment 2089758002.

Don't forget the script I wrote! It's very useful. By default it does not commit automatically but there is an option to allow it to do so. It even asks if you're on the right branch.

• Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

Post Great Fork Merge TODOs

Once the Great Fork Merge occurs and the official IOCCC winner repo and related Official www.ioccc.org web site has been updated, these TODOs need to be performed on the official IOCCC winner repo:

• Close issue Question: Issues that aren’t really major issues but are still issues #5

I've enjoyed this one though I must say! A lot of good things, a lot of fun, a lot of laughs and stories and other things ... thanks for the privilege and honour!

• Transfer any remaining issues to the official IOCCC winner repo

Good idea.

• Mark the temp-test-ioccc repo read-only

From temp-test-ioccc repo settings click on the ((Archive this repository)) button.

THANK YOU for going for that (as I suggested) rather than deleting it.

• Verify site links work
  Ah yes .. that'll be fun! :-)

Process TBD.

Fix any problems found as needed.

Of course.

At the other repo I left a comment regarding an issue I have (about finishing something we talked about the other day) but I think that and this comment is probably all I'll manage to do today for the contest. But who knows. I might be able to take a look at some things at this repo too. For now I'll be afk (or I intend to be afk soon). I think today though I will mostly be reading when not doing the things I have to do.

[lcn2]

Incomplete TODO

NOTICE: This TODO list will be built over a few days. Until this notice is removed, consider this list to be incomplete.

:-)

=-=

• Revise status.json format.
• Write a new tool to generate the new top level status.html from status.json

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file.
It's funny you should say that .. the generating the file. Do you mean creating the file rather than modifying it?

The content of status.json is still TBD, and so is the tool that will mange it.

=-=

The anticipate completion date for this issue is 2024 March 10 2:30 (2:30AM) Pacific.

Like I said over there mostly it is done: except for the YYYY/README.md files. But I can do those after 10 March if necessary so I have no problem with it being closed.

Sure.

BTW: issue #2006 does indicate that it covers such files, and the title has been changed to say "constructed HTML files".

[lcn2]

• Complete issue Enhancement: add templates for issue forms #858

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

It is a ((*top priority)) issue and has been for a while. It pertains to something that is much needed after the Great Fork Merge when people start considering opening up issues on the revised web site.

=-=

• Complete issue Enhancement: Address XXX lines in top level files #2008

Well some of this cannot be done until the submit server is ready, right?

We will need to put in some placeholder text that indicates the exact process has not been created, but will be before the IOCCCMOCK is held.

=-=

• Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

As we need to be the one be performing these last steps, you @xexyl do not need to worry about the details.

=-=

• Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

:-)

[xexyl]

• Complete issue Enhancement: add templates for issue forms #858

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

It is a ((*top priority)) issue and has been for a while. It pertains to something that is much needed after the Great Fork Merge when people start considering opening up issues on the revised web site.

Let me reword that. Besides #3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

That idea of whatever I can work on would be useful but if you have an order I could try and work with it.

=-=

• Complete issue Enhancement: Address XXX lines in top level files #2008

Well some of this cannot be done until the submit server is ready, right?

We will need to put in some placeholder text that indicates the exact process has not been created, but will be before the IOCCCMOCK is held.

That all makes sense to me yes. Thanks.

=-=

• Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

As we need to be the one be performing these last steps, you @xexyl do not need to worry about the details.

I guessed that but I wanted to be sure.

=-=

• Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

:-)

:-) :-)

[lcn2]

Let me reword that. Besides #3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .

We home this clarifies @xexyl .

We will be mostly offline for about 2-3 days working on issue #2008 as well as prep actions for this issue.

[xexyl]

Let me reword that. Besides #3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .

I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

We home this clarifies @xexyl .

Hopefully :-) it does .. yes it does - thanks!

We will be mostly offline for about 2-3 days working on issue #2008 as well as prep actions for this issue.

Best wishes! Today I'm not doing anything for the contest I'm afraid. Having a very off day. Tomorrow i'm sure will be better.

[lcn2]

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .
I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

[xexyl]

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .
I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

Thanks. I want to be sure of something before I finish my part of #3 which I'll ask you there. My part left in that is minimal except of course for checking typos etc. in the FAQ and other like files so that's good. But I won't be doing anything today. Depending on when you get back to me on the concern and the answer too I hopefully can start tomorrow.

[lcn2]

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .
I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

Thanks. I want to be sure of something before I finish my part of #3 which I'll ask you there. My part left in that is minimal except of course for checking typos etc. in the FAQ and other like files so that's good. But I won't be doing anything today. Depending on when you get back to me on the concern and the answer too I hopefully can start tomorrow.

Best wishes, @xexyl