Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify exactly which round formats are allowed #693

Closed
lgarron opened this Issue Jan 13, 2019 · 16 comments

Comments

Projects
None yet
6 participants
@lgarron
Copy link
Member

lgarron commented Jan 13, 2019

We occasionally get (and have recently received) clarifying questions about exactly what round formats are allowed. We've specified most of them in 9b, and I've just landed https://github.com/thewca/wca-regulations/pull/692/files, which specifies that 9b enumerates all valid formats. But apparently 9b is missing more (formerly) valid formats.

I'd love it if someone can run stats on

  • which formats have been used for which events,
  • at how many competitions,
  • tallied by year.

This allows us to see exactly which event formats we are "removing", and how much their use has tapered out.

@jfly jfly self-assigned this Jan 13, 2019

@jfly

This comment has been minimized.

Copy link
Member

jfly commented Jan 14, 2019

This gets tricky because combined rounds have 2 phases with different formats, but we only store the format of the second phase. AFAIK, we can only deduce the format of the first phase by looking at results to see how many skipped results there are for people who didn't make the cutoff. I hacked together Ruby script that does this, and prints out a warning if the people who didn't make the cutoff didn't all complete the same number of solves.

# frozen_string_literal: true

require 'set'

RoundKey = Struct.new(:competition_id, :event_id, :round_type_id)
all_results = Result.all # .limit(10000)
results_by_round_key = all_results.group_by { |r| RoundKey.new(r.competitionId, r.eventId, r.roundTypeId) }

def get_only_element!(arr)
  throw "Uh oh" if arr.length != 1
  arr.to_a[0]
end

Round = Struct.new(:competition_id, :year, :event_id, :phase_1_format, :phase_2_format)
rounds = []

results_by_round_key.each do |round_key, results|
  format_id = get_only_element!(results.map(&:formatId).to_set)
  if %w(c d e g h).include?(round_key.round_type_id)
    missed_cutoff, _made_cutoff = results.partition(&:missed_combined_round_cutoff?)
    potential_phase_1_formats = missed_cutoff.map { |result| result.solve_times.reject(&:skipped?).length }.to_set
    # We expect all the people who didn't make the cutoff to have the exact
    # same number of unskipped solves. If they don't, there's something weird
    # going on, and we can't deduce the format of the first phase of this
    # combined round.
    if potential_phase_1_formats.length > 1
      url = "https://www.worldcubeassociation.org/competitions/#{round_key.competition_id}/results/all#e#{round_key.event_id}_#{round_key.round_type_id}"
      puts "Weird. Check out #{url} when you get a chance"
      next
    end
    phase_1_format = get_only_element!(potential_phase_1_formats).to_s
    phase_2_format = format_id
  else
    phase_1_format = format_id
    phase_2_format = nil
  end

  year = /[0-9]{4}$/.match(round_key.competition_id)[0].to_i
  rounds << Round.new(
    round_key.competition_id,
    year,
    round_key.event_id,
    phase_1_format,
    phase_2_format,
  )
end

counts = rounds.group_by { |round| [round.year, round.event_id, round.phase_1_format, round.phase_2_format] }
               .map { |key, items| [key, items.count] }

puts(counts) # <<<

To run this script, clone the WCA website repo, and place this script in the WcaOnRails directory in a file called identify_formats.rb. You can run it by doing:

$ bin/rails runner identify_formats.rb

Unfortunately, it's quite slow (I think it takes 8-9 minutes on my laptop). @lgarron, hopefully this is enough for you to push this over the finish line! I'm timing out for the day.

@Jambrose777

This comment has been minimized.

Copy link

Jambrose777 commented Jan 15, 2019

@SAuroux has a similar script for this counting the number of blank results in a round. Maybe that can help here.

@timreyn

This comment has been minimized.

Copy link

timreyn commented Jan 15, 2019

https://pastebin.com/NXNYKrda

333 | a | 5 means average of 5, and someone did the full 5 solves.
333 | a | 3 means average of 5, and someone did 3 solves and then didn't make the cutoff.

@Jambrose777

This comment has been minimized.

Copy link

Jambrose777 commented Jan 15, 2019

It's worth mentioning that there are some rounds that were investigated and have been manipulated to "make sense" a lot of the events like "event | a | 3" came from this.

@lgarron

This comment has been minimized.

Copy link
Member Author

lgarron commented Jan 16, 2019

Thanks!

I'm going to revert ea29e09 (Clarify that 9b specifies the valid combined rounds.) until we have this figured out.

@lgarron

This comment has been minimized.

Copy link
Member Author

lgarron commented Jan 20, 2019

lgarron referenced this issue Jan 31, 2019

Get rid of BoX formats in 9b1a). (#671)
Get rid of BoX formats in 9b1a).

Fixes #658.

@lgarron lgarron added the clarity label Jan 31, 2019

@lgarron

This comment has been minimized.

Copy link
Member Author

lgarron commented Feb 13, 2019

@thewca/wrc-team: I'd like to send a PR to crystallize the following formats as official, with formats boxed in this screenshot:

screen shot 2019-02-13 at 12 20 01

I think the only controversial ones might be:

  • Bo2 for Multi-Blind.
    • Mainly since Bo2 is actually more common than Bo3.
    • Keeping only Bo1/Bo3 would have the nice effect that every event has exactly one "full" format, and one "truncated" format.
  • No Bo2 for Fewest Moves
    • This is kind of inconsistent with Multi-Blind. I presume this is uncommon because FMC has a mean, while multi does not – so the step from 2 to 3 hours is more "worth it".
  • No combined rounds for Fewest Moves.

In general, I would love to see us move away from Mean of 3, due to the harsh impact of a DNF in those events. However, that's out of the scope of this change.

Do the choices in the screenshot look reasonable?

@lgarron

This comment has been minimized.

Copy link
Member Author

lgarron commented Feb 13, 2019

@thewca/software-team: any thoughts/concerns about simplifying the list of formats as described in the previous post?

@Jambrose777

This comment has been minimized.

Copy link

Jambrose777 commented Feb 14, 2019

FMC Combined Rounds: I agree that this makes little sense. If a competition can support x people in the room for an attempt it takes no extra resources (besides paper) to support x people in the next attempt. This decision seems great to me!

FMC bo2: The 6 competitions with this in 2018 were:

  • MundetOpen2018 5 DNS, 18 Competitors
  • Murcia2018 4 DNS, 13 Competitors
  • NoPlaceForDNFSpring2018 8 DNS, 18 Competitors. From Report "We ended up compromising on 2 attempts, which, in retrospect is silly (some people just showed up for 1 attempt because they wouldn't be able to get a mean anyway). In the future, if I delegate a competition with FMC, I will either have best of 1 or mean/best of 3."
  • BasauriOpen2018 0 DNS, 7 Competitors
  • Tlaxcala2018 10 DNS, 41 Competitors
  • CUBIFMA2018 2 DNS, 9 Competitors
    Also interestingly, 5 of these competitions were Spain / Latin America (and the other being US). I seem some merit in keeping this format since it is an event which requires a dedicated hour of resources and the jump from 1 to 3 hours is a lot. However, I also see the issue which was pointed out in the report of it's a bit pointless to hold and the resources can be better spent for another event at a competition and make the FMC round fairer. Overall, I'm also in favor of removing this possibility for this reason and that it just isn't used that much.

MBLD bo2: I see a lot of merit in keeping this due to the resources needed to hold the event (this is the hardest event to hold organizational-wise on a large level).

MBLD Combined Rounds: I have to disagree with this decision. As stated above, MBLD is the hardest event to hold as an organizer. In order to accommodate a large amount of competitors the staff for the competition have to be dedicated, as well as a large amount of time for the event (generally speaking it will take longer than 1 hour for an average competition). For larger events, using a cutoff allows, for example, dedication a large amount of time and staff for the event for the first attempt while having the other two attempts only be for a small number of competitors. This allows more people to compete while not causing a detriment in the amount of time used from the competition. So, I would really like to see the combined formats stay for multi-blind.

@lgarron

This comment has been minimized.

Copy link
Member Author

lgarron commented Feb 14, 2019

MBLD Combined Rounds: I have to disagree with this decision. As stated above, MBLD is the hardest event to hold as an organizer. In order to accommodate a large amount of competitors the staff for the competition have to be dedicated, as well as a large amount of time for the event (generally speaking it will take longer than 1 hour for an average competition).

Hmm, the way we're calculating stats shows only one combined round for multi. The earlier accidental removal shows a few more examples: #690 (review)

However, our approach might have lots of false negatives. Do you have a way to find/calculate how often this is actually used in practice?

@Jambrose777

This comment has been minimized.

Copy link

Jambrose777 commented Feb 14, 2019

MBLD Bo2 with Bo1 Cutoff:

Competition # of Competitors Cutoff
Euro2012 17
Euro2014 28
Euro2016 33
WC2011 17

MBLD Bo3 with Bo1 Cutoff:

Competition # of Competitors Cutoff
CzechOpen2014 5
KievSuperBlind2019 9

MBLD Bo3 with Bo2 Cutoff:

Competition # of Competitors Cutoff
NordicChampionship2018 17
WC2017 56

Queries:
SELECT competitionId, COUNT(DISTINCT personID) FROM Results WHERE eventId = '333mbf' AND formatId = '2' AND value2 = 0 GROUP BY competitionId, roundTypeId
SELECT competitionId, COUNT(DISTINCT personID) FROM Results WHERE eventId = '333mbf' AND formatId = '3' AND value2 = 0 GROUP BY competitionId, roundTypeId
SELECT competitionId, COUNT(DISTINCT personID) FROM Results WHERE eventId = '333mbf' AND formatId = '3' AND value2 <> 0 AND value3 = 0 GROUP BY competitionId, roundTypeId

So as you can see, its generally only the big competitions that have utilized this, which makes sense why the number of times utilized is low since we have so few of them.

@Jambrose777

This comment has been minimized.

Copy link

Jambrose777 commented Feb 14, 2019

Just noticed: I’m also fine completely removing the cutoff for 3x3. Its is always recommended not to use, and hasn’t ever caused an issue per say. I’ll run some stats tonight for 2x2, pyraminx and skewb as those events are recommended not to have a cutoff (not as strongly as 3x3) and the cutoffs typically provide no value unless hey are extremely low. So maybe some other restriction on this would be good?!?

@viroulep

This comment has been minimized.

Copy link
Member

viroulep commented Feb 16, 2019

I agree with Jacob about the cutoff for Multi: it's use very rarely but in these case it was pretty useful.
It's worth noting we could have had a much stricter qualification limit for WC2017 (it was >= 5 points, cutoff was >= 10 points), but we wanted to accommodate as many competitors as possible and we were not able to accommodate the 90 competitors registered on Saturday because of staff constraints.

I think we need to keep the cutoff for 2x2/Pyraminx/skewb.
Again in big competitions you could want to keep these events open to everyone (ie: no qualification time), yet you have not infinite time and cutoff is a reasonably reliable tool to make sure X competitors only do 2 solves (to use the WC2017 example, we needed to have 300 competitors be done with 2x2 in 30 minutes, we couldn't have done this without setting the cutoff appropriately; the same applies to Euro2018, and will most likely apply to WC2019).

@lgarron

This comment has been minimized.

Copy link
Member Author

lgarron commented Feb 16, 2019

Okay, I'll send a PR that conservatively keeps:

  • FMC:
    • Bo1
    • Bo2
    • Bo1/Bo2
    • Mo3
    • Bo1/Mo3
    • Bo2/Mo3
  • Multi-Blind:
    • Bo1
    • Bo2
    • Bo1/Bo2
    • Bo3
    • Bo1/Bo3
    • Bo2/Bo3
  • 3BLD, 4BLD, 5BLD:
    • Bo3
  • Everything else:
    • Ao5
    • Bo2/Ao5

lgarron added a commit that referenced this issue Feb 17, 2019

@jamesquinn1

This comment has been minimized.

Copy link

jamesquinn1 commented Feb 19, 2019

@lgarron
mbld mo3? assuming that being added was just a mistake.

@lgarron

This comment has been minimized.

Copy link
Member Author

lgarron commented Feb 19, 2019

@lgarron
mbld mo3? assuming that being added was just a mistake.

Are you referring to ddb6a66#r32347538 ? That's already fixed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.