Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question about some Idiosyncrasies of network promotion and related policy #1229

Closed
hydrogenpi opened this Issue Apr 21, 2018 · 28 comments

Comments

Projects
None yet
@hydrogenpi
Copy link

hydrogenpi commented Apr 21, 2018

Question about some Idiosyncrasies of network promotion and related policy
http://archive.is/aq1iS

**_TLDR version ==
{
Yesterday evening we had a new network (599bb7337) that should have been promoted, but due to whatever reasons (possibly a bug etc) it wasn't promoted. However, it came very close to passing 55% and in fact was much closer than the official network 125 that was promoted at 53%. It got the "PASS" three times when total match games had exceeded 400 played. By all accounts this new network should be network 126.

Additionally, it appears network 599bb7337 has vastly improved in laddering and endgame skills and it a significant improvement. Some have already posted videos on youtube about this.

Eventually a new winner will naturally appear and be the official 126 net if the 599bb7337 isn't manually promoted and nothing else is done. The real question is what if like network 125 it settles at a lower wr% than say what 599bb7337 settled on, AND it wasn't as good at ladders compared to 599bb7337. And it took another week. Now that would be a real pitty.

My proposal is to wait and see which network naturally gets to the #126 mark and before OFFICIALLY promoting it as network 126, due to the UNIQUE circumstances in this case, do a final test match Between the two networks and whichever one ends up on top gets to get crowned as 126. (especially if the would-be official network 126 actually ends up lower wr% than 599bb7337 then we really need a "recount" ) Otherwise we are not being fair at all to 599bb7337. If strength is the true goal, and the decision is to wait it out anyway, and the time has been spent as a sunken cost anyhow, then why not pick the best to go forward, esp. since 599bb7337 seems so much better at ladders?
}_**

This is not a rehash of anything I might have posted or brought up before. To be clear, I am not advocating for any sort of ‘fake’ promotions of new networks nor is this some sort of “hurry up and promote because we are getting impatient” sentiment. Previously, I recall, the main developer himself @gcp had stated on github or elsewhere that in essence he doesn’t just want “PASS” but he wants “stronger” networks. I agree and below is my perspective and interpretation of this.

I believe there is relevant context and that the sentiment and question that I’m bringing up both comports with the spirit of desiring true strength instead of syntaxical/lexiconal “PASS” and is in alignment with the broader more encompassing goals of the project at large.

Allow me to illustrate just one brief example, or two. Network #125 promoted over #124 with 53.46% winrate with 232 wins against 202 lossess… on or about 2018-04-18; While a mid 53% wr is still very respectable, it is by no means a bona fide 55%. The only reason it promoted was because it hit the critical criteria for a “PASS” (albeit also momentarily) somewhere before its total of 434 match games and then subsequently steadily fell downwards until it settled at its 53.46% final wr.

Tonight, candidate contender for Network #126 (59bb7337) reached above 55% gating threshold when it was already over 400 total match games played and several times hit the “PASS” threshold and triggered a “PASS” registered on the website. In point of fact, it already hit and then re-hit the 55% threshold and registered as “PASS” for many times (I observed at least three times) AFTER it had already hit the 400 game mark.

It is safe to say that the math shows that in terms of raw strength gains, based on the numbers and the same objective metrics and standards that we have always used, that the candidate contender for Network #126 (59bb7337) is stronger (when objectively compared based on raw numbers and statistical math) than the Network #125 that was actually promoted (e8601c38) with only a 53.46% wr.

A second point, network #125 (e8601c38) was promoted with only 53.46% wr whilst an earlier contender/candidate network (db68982c) of the same generation/eon was rejected even though it had a higher winrate at 54.12% and was also well over the 400 total match game threshold once it settled on that higher winrate than the network #125 (e8601c38). Now my personal stance is that in a case like this, when going for max true strength, it is the 54.12% network that should have been actually promoted and not the 53.46% that was officially promoted. (how to go about effectuating that is another story, but I would simply say whenever a promote is triggered that then drops below 55%, before officially promoting look to see if any other previous contender networks of that generation had a higher wr% at equal or greater games played and if so, swap!) In this case, the reason one network was promoted over the other had not so much to do with strength but with how the promotion system was set up that if it was by luck it hit 55% first then it “got in” whilst a network that was actually stronger (but never hit 55%) was discarded. This doesn’t make sense to me if the purported goal is to promote based on true strength.

But more controversial perhaps is the case that we have tonight when indeed when we can compare side by side the circumstances of the situation, when we have BOTH the example of Network #126 (59bb7337) getting that official “PASS” registered and hitting above that 55.00% gating threshold exactly like it’s would-be predecessor network #125 (e8601c38) had done so before it, and the similar parallel situation and circumstance whereby the candidate contender for Network #126 (59bb7337) stayed at above the required 55.00% winrate threshold whilst its total match games played were already above the 400 mark.

So begs the question if strength gains is the goal, why is it that Network #125 promoted but network 59bb7337 did not promote? It seems like by each and every standard and metric that is used it should have promoted. In terms of strength gain, it is indeed based on the numbers and the math (which is all anyone can objectively go by anyway) that it is actually stronger (in terms of percentage ratio gain in raw strength) when comparatively compared to its predecessor network #125, and exactly like predecessor network #125, it too also hit the required threshold after it already amassed 400 games or more, and it too went back below the 55% threshold before it reached its final last match game played.

The only difference is that it actually finally landed at a HIGHER win rate % than network #125, but this only makes the case for promoting network 59bb7337 even more persuasive and convincing, not less. So by all appearances (unless I missed something substantial) it would seem that if we were to allow situations like this to stand, that the only thing that is the arbitrating difference that makes a difference in close cases like these isn’t so much raw strength but merely very arbitrary “luck of the draw” type of scenarios in which actual strength is sacrificed for the sake of accepting and entertaining a certain level of imperfect Idiosyncrasies of network promotion and related policy.

And if that is the policy that is fine, it’s just that it doesn’t seem to align nor comport with the stated policy of promoting based on strength and at least to me appears to be inconsistent with the fact that weaker networks have gotten promoted when under exact same circumstances arguably stronger networks have been rejected.

@hydrogenpi

This comment has been minimized.

Copy link
Author

hydrogenpi commented Apr 21, 2018

Looks like this inconsistent policy has already caused all sorts of confusion:

video: https://archive.org/details/LatestNetworkVersion59bb7337LaddersTestNoSound

http://archive.is/TlIyq

image

image

image

image

#1180

@hydrogenpi

This comment has been minimized.

Copy link
Author

hydrogenpi commented Apr 21, 2018

I realize my posts are too long so I'll quickly add addendum to my original thoughts by posing the question of what gains are there to be made by this sort of waiting? To wit: we see for example that after Network 124 was promoted very shortly thereafter in fact within hours there was a 54%+ candidate network that was rejected and yet ultimately what was promoted was a 53.4 wr% and it was several days later AFTER the 54%.

An apt analogy would be when we compare networks of different arch sizes we don't just compare winrates but also we compare the time difference in order to get true strength over time parity. Like for example even though some of the 20 block networks got 80% winrate over the official 10blocks, they weren't in fact stronger because their 5x slower didn't make up for it. etc

On the surface, when both networks are over 400 games, the 54%+ network is already more or less stronger than the 53.4% network, but when taken into account the "time difference", ie that the 54% network came into existence days EARLIER than the 53% network, then the question is obvious one, what exactly have we GAINED by waiting extra few days (lost time) for what is argueably a weaker network even on time parity?

Put another way, there is no doubt that Leela Zero is already pro strength or above and that it is very strong. I think it is fair to say that right now it is still an open question of theoretically what is its ultimate cap/ ceiling. However we do know it has a ceiling (or whatever one calls it) and that as it begins to approach that ceiling, improvement and progress will be slower and slower. This was the same even for the mighty AGZ and the natural course of things. There had been talks in the past about either lowering gating or getting rid of gating altogether, and while I'm a proponent of the former and against the latter, this thread/post isn't about advocating or arguing the change of gating threshold one way or another. But I would like to point out that as LZ gets stronger and stronger, its returns/margins will likely be more and more granular. So these little bits of lost strength (ie rejecting the slightly stronger network in favor of accepting a slightly weaker network merely because based on the so-called luck of the draw) will become more and more prevalent and more magnified into the future. My first job ever was sacking bags at Kroger when I was 16 and I still recall the store manager preaching to us that when the retail store only makes a 1% profit margin, it can’t afford a 3% shrink or loss due to customer or employee theft. Ironically, Leela Zero is strong enough on its progression curve that I don't think it can afford these sort of inefficiences and setbacks stemming from Idiosyncrasies anymore.

As it stands, it appears that the 54% (ie what would have been Network 126) was indeed rejected and not promoted.... So, moving on from there, discussing the forward future, I would say that it seems very possible that perhaps say in the next few hours or next coming days we might eventually get an official network 126 promoted. But My concern is this, what if just like the network 125 before it, it (the then promoted new net 126) actually in reality ends up getting promoted but has an LOWER winrrate than the 54.64% that we had already gotten the multiple "PASS"es for yesterday evening? This brings us back to my previous point in the above paragraph, and that is what have the project actually gained by the extra hours, days of waiting? When it could have simply accepted the "PASS" of the 54% network when it had first happened?

Now obviously the time in-between isn't completely lost and/or wasted, in that they are used to create self-playing games and thus training data and also thus the more self playing games the stronger the trained networks can hope to be or become.... but wouldn't all of this be even better had the actually stronger network been accepted much earlier on and then used to pump out the higher quality self -playing games? And even if the arguement was that a 53% vs 54% isn't all that much of a difference, but when you've already been pumping out self playings games on the exact same network #125 for going on four or five days now, (as in this case for network 125) wouldn't it make sense for the sake of the health and diversity of the games generation to switch to a different net, especially considering in light of the network that we got last night which was 54+% and reached multiple PASS and maintained 55% and above after it went above 400 games?

To be fair, I have no idea how much more effective progress would happen or would have happened had different strategies been tried in order to get max strength with most efficent manner possible. A while back I proposed a thought about promoting all 53% at 1000 games and 54% at 600 games or thereabouts, and imagined that it would get far faster progress since we were getting tons more at 53%... but that wishful thinking also had to be tempered by the reality that even doing everything possible and not "wasting" (for lack of better terms) a single moment, its inevitable that as the network gets stronger, it will progressively become harder and harder to get incrementally progress.

@roy7

This comment has been minimized.

Copy link
Collaborator

roy7 commented Apr 21, 2018

So ultimately the question is, is there a bug in promotion of matches when the match has already left the queue? We have a special case to test it here. Just looking at it, I don't see why this would fail. It's the same logic/test that is used to decide if a PASS appears? The test === true. Maybe @Dorus can take a peek.

After sending the last match out for this network (425 of 400) it triggered the "Early fail pop" section of the code to remove it from the pending list. If wins arrive after removal it should still promote though given the section of code linked above.

@Marcin1960

This comment has been minimized.

Copy link

Marcin1960 commented Apr 21, 2018

Why not to promote networks manually?

55% threshold is for 400 matches, for 430 it should be less than 55%. Isn't what math and logic say?

@Ka-zam

This comment has been minimized.

Copy link

Ka-zam commented Apr 21, 2018

Will it matter so much in the long run?

Just lower the bound to 54.5% then and you will have more promotions. And still there will be cases where some network got 54.47% and therefore it should be manually promoted anyway.

@hydrogenpi

This comment has been minimized.

Copy link
Author

hydrogenpi commented Apr 21, 2018

@Ka-zam I don't suppose it will have an effect on the long run max potential of LZ purely from a technicial standpoint. However, couple days lost here and there can quickly add up over time... and with PR stunts like the one that Golaxy pulled, yes it would matter in the court of public perception and hence indirectly in terms of influx of clients or ability to maintain client numbers which in turn for a distributed crowd sourced project like this, could have significant meaningful impact, however indirectly.

@Ka-zam

This comment has been minimized.

Copy link

Ka-zam commented Apr 21, 2018

Perhaps we should look for promoted networks that reached 55+% @400 games and if they subsequently slipped below as the number of games went above 400 they should be demoted instead?

They are spreading a disease in the selfplay body! :)

@hydrogenpi

This comment has been minimized.

Copy link
Author

hydrogenpi commented Apr 21, 2018

Sarcasm aside, I personally am not suggesting what you mentioned, I'm advocate for promote all that is better, ie 55% at 400, 54% at 600, 53% at 1000 etc you get the point. Even if it was just 400 tests at 50.1% gating, on average in general over the long run the false positives and false negs will cancel each other out, so no need to do insane # of games like for example 10000 match games or more etc. Now of course I think this would be better than no gating, or even accepting a -50 elo gating like the Chess guys did that.

TLDR as the LZ network gets stronger its unreasonable to expect that we will hit 55% with anywhere near the same frequency, and the unwillingless to accept lower quantum levels (more grandular wins) creates the environment whereby there could be significant time loss encapsulated within pockets of temporal regions that otherwise would have been unlocked had there been a system in place to accept and partake in all such minor victories instead of waiting steadyfast for the ever less probabalistic lottery. Simple thought experiment, pretend we set arbitrary gating to 70%.... I don't think anyone believes we wouldn't needlessly get stuck months and months without a new network.

@Dorus

This comment has been minimized.

Copy link

Dorus commented Apr 21, 2018

I realize my posts are too long

Yes they are. Much too long. Please try to make them more concise and to the point.

so I'll quickly add addendum

damned, another 940 words.


Indeed, the promoted 53% winrate network is unfortunate. However this is considered an acceptable loss of the speed up (less match games) we get from SPRT. Promoting one 53% network every 200 matches will still make an demotion less likely than accepting all of them. Also remember 53% is still over 50% so it's still likely this network was an actual improvement.

On the other hand, the 54.67% rejected network is strange. I checked the matches and it even got over 55% above 400 games. At that point it have should been promoted. I dont care if we manually promote it now or not, but this is a bug on the server we should look into. @roy7 i'll see if i can discover anything in there. Indeed it should have been promoted.

@kuba97531

This comment has been minimized.

Copy link
Contributor

kuba97531 commented Apr 21, 2018

@hydrogenpi
There is a well working pipeline, which constantly provides progress and you want to monkey patch it every time you dislike some event.

You produce a lot of ideas, opinions and proposals. "I think we should do this". "I think this would be better". "I'm advocating for this" etc.
But it all boils down to (all quotes are yours) "I have no idea how much more effective progress would happen or would have happened had different strategies been tried"

I will tell you a secret that most people contributing to this project can give you out of the top of their heads thousands ideas that might potentially improve the efficiency of the pipeline or strength of the program.

Spiting mindlessly ideas that come to your mind has absolutely 0 value (it actually has negative value because some people will read it).
What has value is either some solid mathematical proof or convincing data / experiment results.

@hydrogenpi

This comment has been minimized.

Copy link
Author

hydrogenpi commented Apr 21, 2018

@kuba97531 the experiment you talk of would in fact require the likes of hundreds or thousands of clients in order to conduct any such test. As you well know, just because code, training data and etc are open source does not mean any individual can just do the types of experiements that only the project developer himself @gcp and others who ultimately steer the direction of the ship in terms of LeelaZ mainline can do. You should know that, as I assume most people understand that.

There obviously would be an improvement if "PASS" networks like the one from last evening WERE promoted on site (as it should have been) as opposed to waiting and perhaps wasting more time just to days later promote another network that , --as has shown via previous examples that actually happened--, may or may not even be stronger than the one that was passed on by and rejected days earlier. Realising of course I was not in a position to provide data (realistically no one other than gcp and perhaps Google would be) on just how much of an improvement we would actually see (quantifiable etc) I qualified my remarks by disclaiming that I was unsure as to the amount of gap or overlap between inefficiencies that could be recovered by resolving such known and proven Idiosyncrasies vs when it would hit against the natural asymptote of progression even when under the most ideal situations. That admission of not knowing exactly where on the scale or spectrum such adopted improvements would land does not in and of itself forfeit and is not a waiver of my ability to share my opinion that the way networks are promoted should or could be optimized and improved based on the factual inconsistencies that we have already witnessed. There is no doubt there would be improvements, the only question was whether or not it was enough of an improvement in terms of margin size to merit the human resources involved and to overcome the natural coefficent of friction in order to push past any inertia to effectuate an implementation.

Resolving such Idiosyncrasies would only serve to help the progress of the overall project, and would not do it any harm. No one has suggested any "monkey patch", however even if policy remains the same, additional clarification would go a long way towards resolving confusion (see the videos posted already with the network 126 from last night, and no these were not my videos) or at the very least serve to help set proper expecations and /or even temper any doubts and also act as a framework of consistent guildlines and measurements tools for the rest of the community. Again, even if nothing is changed in code or policy, any official or additional clarification on this front could only help the project, not harm it.

You don't require "solid mathematical proof" nor "convincing data / experiment results" to be able to logically ascertain that passing and rejecting a 54.64% network (that had actually gained the "PASS" sign three times no less, even after it reached 400 games) only to days later accept and promote a 53% network (weaker network that just so happened to get in based on faster returns of won games earlier on etc) is an overall detriment to the general progress of the project both in terms of time and strength...

@Friday9i

This comment has been minimized.

Copy link

Friday9i commented Apr 21, 2018

Diversity in selfplay games is important (that's why there is some random choice of moves in the first 30 moves). Hence, the more I think about it, the more "always promote" seems attractive... And if we don't want to go that far, accepting more easily new networks (1 sigma instead of 2 sigma, so around 52.5% threshold?) would ensure significantly more diversity in selfplay: even if some new networks are not stronger, that would be useful.
One question: is it possible one way or another to modify match games in order to include them in the selfplay data? That would allow longer matches from time to time without slowing selfplay

@hydrogenpi

This comment has been minimized.

Copy link
Author

hydrogenpi commented Apr 21, 2018

@Friday9i since at any given moment around 10 to 20 percent of all games played are match games and they are of higher quality (more MCTS rounds etc) incorporating it might help with both diversity and speed of progression/learning and plus the fact that match games are argueably stronger can't hurt either.

In an always promote scheme, what is the benefit to allowing weaker (less than 50%) networks to promote? Is it because of the thought it might dig itself out of a local max? If that is the line of thinking, since the LZ project has been at 55% gating throughout the entire history of its training so far, would allowing a full always promote (vs a partial always promote that sets min gating at 50%) really even help that much? If it has already entrenched itself into a local maximum it would be pretty difficult to get back out unless folks are willing to accept hundreds of ELO's regression.

@Friday9i

This comment has been minimized.

Copy link

Friday9i commented Apr 21, 2018

Yes in my understanding it is to go out of a local max, to explore more broadly the landscape and find new "directions" to other local max, possibly much better. Moreover, it generates also some mistakes from weaker networks, which is useful to learn not to repeat these mistakes. All together, more diversity, more opportunities to explore and learn new things.

@Dorus

This comment has been minimized.

Copy link

Dorus commented Apr 21, 2018

@kuba97531 Please stop ranting, it's bad enough with one guy doing it.

You produce a lot of ideas, opinions and proposals. "I think we should do this". "I think this would be better". "I'm advocating for this" etc.
But it all boils down to (all quotes are yours) "I have no idea how much more effective progress would happen or would have happened had different strategies been tried"

Really, why did i spend a minute reading this? This adds zero to the discussion. Fresh ideas are always welcome and we should be open to discuss them. I'm not saying we should pick up any random idea we get (that's impossible, many are contradictions of each other).

I will tell you a secret that most people contributing to this project can give you out of the top of their heads thousands ideas that might potentially improve the efficiency of the pipeline or strength of the program.

Spiting mindlessly ideas that come to your mind has absolutely 0 value (it actually has negative value because some people will read it).
What has value is either some solid mathematical proof or convincing data / experiment results.

Serious, take your trolling to reddit please. On here you should come with actual arguments.

@kuba97531

This comment has been minimized.

Copy link
Contributor

kuba97531 commented Apr 21, 2018

@Dorus
Fair points. I appologize.

@hydrogenpi

This comment has been minimized.

Copy link
Author

hydrogenpi commented Apr 21, 2018

@kuba97531

I also apologize for some of my long windness and also if rhetoric was used... With this last input I've said all I ever need to say on this subject and whatever happens happens. I'll state that if the decision is to wait for a new 126 network and it lands/settles at less than the network (599bb7337) final winrrate and it doesn't appear to be as good as ladders as network (599bb7337) then it would be self evident that if no final "recount" is done in a head to head match for both networks that both contend for the 126 crown, that indeed despite what is said, the actions are so that the project is not in fact putting true strength first and is not in fact promoting based on strength.
**_TLDR version ==
{
Yesterday evening we had a new network (599bb7337) that should have been promoted, but due to whatever reasons (possibly a bug etc) it wasn't promoted. However, it came very close to passing 55% and in fact was much closer than the official network 125 that was promoted at 53%. It got the "PASS" three times when total match games had exceeded 400 played. By all accounts this new network should be network 126.

Additionally, it appears network 599bb7337 has vastly improved in laddering and endgame skills and it a significant improvement. Some have already posted videos on youtube about this.

Eventually a new winner will naturally appear and be the official 126 net if the 599bb7337 isn't manually promoted and nothing else is done. The real question is what if like network 125 it settles at a lower wr% than say what 599bb7337 settled on, AND it wasn't as good at ladders compared to 599bb7337. And it took another week. Now that would be a real pitty.

My proposal is to wait and see which network naturally gets to the #126 mark and before OFFICIALLY promoting it as network 126, due to the UNIQUE circumstances in this case, do a final test match Between the two networks and whichever one ends up on top gets to get crowned as 126. (especially if the would-be official network 126 actually ends up lower wr% than 599bb7337 then we really need a "recount" ) Otherwise we are not being fair at all to 599bb7337. If strength is the true goal, and the decision is to wait it out anyway, and the time has been spent as a sunken cost anyhow, then why not pick the best to go forward, esp. since 599bb7337 seems so much better at ladders?
}_**

@l1t1

This comment has been minimized.

Copy link

l1t1 commented Apr 22, 2018

a new star ee62 is rising, maybe you can let it play with 599b let head to head by yourself @hydrogenpi

2018-04-22 05:44 ee62c0a3 VS e8601c38 50 : 40 (55.56%) 90 / 400

@Friday9i

This comment has been minimized.

Copy link

Friday9i commented Apr 22, 2018

Nope, but surprisingly, 59bb officially became the new king LZ126 (after more than 24 hours as "fail")!

@l1t1

This comment has been minimized.

Copy link

l1t1 commented Apr 22, 2018

@Friday9i it's a good news

@l1t1

This comment has been minimized.

Copy link

l1t1 commented Apr 22, 2018

2018-04-21 00:11 59bb7337 VS e8601c38 245 : 199 (55.18%) 444 / 400 PASS
the server must send some new games to the match,

@billyswong

This comment has been minimized.

Copy link

billyswong commented Apr 22, 2018

@l1t1 @Friday9i If you look at those last 5 "late submitted" games, they are all from the same client "92", all saying the new net win, and the dates are inconsistent. Some are on 2018-04-21 and some later are on 2018-04-20.

It smells suspicious to me.

@roy7

This comment has been minimized.

Copy link
Collaborator

roy7 commented Apr 22, 2018

Yes, someone uploaded false results to force that promotion. It's very frustrating.

@Yakago

This comment has been minimized.

Copy link

Yakago commented Apr 22, 2018

Looking forward to the next discussion about how 3f6c should be manually promoted since 59bb was not legitimately promoted and how the 5 games have actually sabotaged the project since 3f6c is so much stronger

@roy7

This comment has been minimized.

Copy link
Collaborator

roy7 commented Apr 22, 2018

Haha that discussion has already started on Discord. :)

@tterava

This comment has been minimized.

Copy link
Contributor

tterava commented Apr 22, 2018

All of this is so unnecessary. Can't we just fix the bug where networks don't necessarily promote even if it shows PASS and call it a day?

@roy7

This comment has been minimized.

Copy link
Collaborator

roy7 commented Apr 22, 2018

@tterava Yes we are, there's already a pull request to do so. The bigger issue is what happens if people keep doing this? We need to spend some programming time now to harden the server up some instead of working on other things.

@hydrogenpi

This comment has been minimized.

Copy link
Author

hydrogenpi commented Apr 23, 2018

Everyone, I take back most of what I said in long paragraphs, after finding out via testing that ( (599bb7337) is not some magically network that suddently gets rid of all ladder issues, as I've found in some cases it loses embarrasing ladders to Leela 11.

Imagine my suprise this evening when I check back online and apparently Andy Liu played a 2H against Leela Zero and on network 127 no less. He sang many praises of Leela Zero, and Andy Liu has won against Haylee in the past and seems stronger than her, so when Andy stated that Leela Zero is basically superhuman already, I'm inclined to believe it.

Because I had been so vocal about passing (599bb7337) I do hope no one thinks whatever happend with the malicious forced promote had anything to do with me, I will affirmatively state I have no knowledge, involvement, association, etc.

Yes I still do think some fundamental changes needs to be made with regards to accepting more granular progress in order to avoid as much time wasting inefficencies as possible as the project gets stronger and stronger... but that's pretty much it that pretty soon everyone will share the sentiment so there is no more use for my original thread and I'll close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.