Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleanup jira's concept of 'master' and '6.0' [LUCENE-7271] #8326

Closed
asfimport opened this issue May 4, 2016 · 22 comments
Closed

Cleanup jira's concept of 'master' and '6.0' [LUCENE-7271] #8326

asfimport opened this issue May 4, 2016 · 22 comments
Assignees
@hossman
Labels
legacy-jira-priority:Major legacy-jira-resolution:Fixed type:bug

Comments

@asfimport
Copy link

asfimport commented May 4, 2016 •
edited

Jira's concept of "Fix Version: master" is currently screwed up, as noted & discussed in this mailing list thread...

http://mail-archives.apache.org/mod_mbox/lucene-dev/201604.mbox/%3Calpine.DEB.2.11.1604131529140.15570@tray%3E

The current best plan of attack (summary) is:

  • use Jira's "Merge Versions" feature to merge master into 6.0
  • add a new master (7.0) to use moving forward
  • manually audit/fix the fixVersion of some clean up issues as needed.

I'm using this issue to track this work.

Detailed Check list of planned steps:

  • S1: Generate a CSV report listing all resolved/closed Jira's with 'fixVersion=master AND fixVersion=6.1'
  • S2: Generate two CSV reports containing all issues that match these 2 queries for fixVersion=master and fixVersion=6.0
    *** master: https://issues.apache.org/jira/issues/?jql=project%20in%20%28LUCENE%2C%20SOLR%29%20AND%20fixVersion%20%3D%20master%20ORDER%20BY%20key%20DESC
    *** 6.0: https://issues.apache.org/jira/issues/?jql=project%20in%20%28LUCENE%2C%20SOLR%29%20AND%20fixVersion%20%3D%206.0%20ORDER%20BY%20key%20DESC
    ** these reports can be attached to this issue (LUCENE-7271) for posterity in case people want to later review what the state of any issue was before this whole process was started and versions were changed/merged
  • S3: Use Jira's "Merge Versions" feature to merge "master" into "6.0"
    • This needs to be done distinctly for both LUCENE and SOLR
  • S4: Add a new "master (7.0)" version to Jira
    • This needs to be done distinctly for both LUCENE and SOLR
  • S5: audit every issue in the CSV file from S1 above to determine if/when the issue should get fixVersion="master (7.0)" added to it or fixVersion="6.0" removed from it:
    • if it only ever had commits to master (ie: before branch_6x was made on March 2nd) then no action needed.
    • if it has distinct commits to both master after branch_6x was made on March 2nd, then fixVersion="master (7.0)" should definitely be added.
    • if it has no commits to branch_6_0, and the only commits to branch_6x are after branch_6_0 was created on March 3rd, then fixVersion="6.0" should be removed.
    • if there are no obvious commits in the issue comments, then sanity check why it has any fixVersion at all (can't reproduce? dup? etc...)
  • S6: Audit CHANGES.txt & git commits and replace fixVersion=6.0 with fixVersion="master (7.0)" on the handful of issues where appropriate in case they fell through the cracks in S5:
    • Look at the 7.0 section of lucene/CHANGES.txt & solr/CHANGES.txt for new features
      • currently only 1 jira mentioned in either files in 7.0 section
    • Use git co releases/lucene-solr/6.0.0 && git cherry -v master | egrep '^\+' to find changesets on master that were not included in 6.0
      • currently ~40 commits
    • before removing fixVersion=6.0 from any of these issues, sanity check if this is a deprecation type situation (involving diff commits in both 6.0 and master) in which case fixVersion="master (7.0)" should be added in addition to fixVersion=6.0

Migrated from LUCENE-7271 by Chris M. Hostetter (@hossman), resolved May 11 2016
Attachments: jira_export.pl, LUCENE-7271_S1_report.csv (versions: 2), LUCENE-7271_S1_report.xls (versions: 2), LUCENE-7271_S2_6.0_report.xml, LUCENE-7271_S2_master_report.tgz, LUCENE-7271_S5_hoss_todo.txt, LUCENE-7271_S6_report.txt
Linked issues:
@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

I haven't seen any objections to this plan, and i haven't been able to think of any flaws or possible improvements.

My plan is to start working through these steps tomorrow (Friday) morning ~9AM my time, (~1600 UTC)

Steps S1-S4 will need to be done carefully and in a single block to reduce the risk of missing issues edited between steps (but obviously skimming mail for issues people modify during that window can be done after the fact).

Steps S5 & S6 should be done ASAP after that to reduce confusion as people read/edit jiras, but don't need to be rushed (ie: i'll go to lunch at some point) and can be divided up among multiple people if other folks want to volunteer.

I'll track progress with comments here, and attach the reports i generate from S1, S5, and S6 to this issue as i go.

I'll be on freenodes #lucene IRC channel the whole time if people have concerns or want to coordinate on helping out with S5 & S6.

@asfimport
Copy link
Author

Anshum Gupta (@anshumg) (migrated from JIRA)

LGTM Hoss. There's on thing that I don't have clarity on though.

S5: if it only ever had commits to master (ie: before branch_6x was made) then no action needed

How would this be ever true, considering that the list that you'd generate would have fix version as master AND 6.1?

I would like to help but I'm not sure if I'd be available as I'm traveling for Apache Big Data tomorrow. I'll try and sync up on IRC if I can help.

@asfimport
Copy link
Author

Michael McCandless (@mikemccand) (migrated from JIRA)

Thank you @hossman!

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

How would this be ever true, considering that the list that you'd generate would have fix version as master AND 6.1?

It shouldn't ever be true, and it might be paranoied overkill on my part, but ultimately i'm just trying to cover all the basis in terms of "this issue has 'master' in fixVersion, what was the intent when that was added?"

Imagine this hypothetical timeline...

  • master and branch-5x are the only active branches
  • commit#1 for JIRA-XXX is commited to master
  • JIRA-XXX is marked fixVersion=master and resolved
  • branch_6x is created
  • branch_6_0 is created
  • a bug is noticed in the new functionality added by JIRA-XXX and the issue is re-opened
  • a fix is created and commit#2 is made to master & backported to both branch-6x and branch_6_0
  • JIRA-XXX is updated to include fixVersion=6.1, but the dev doesn't notice that 6.0 is missing and doesn't think to add for some reason
  • we merge master -> 6.0, and now the issue only lists fixVersion="6.0, 6.1" w/o listing "master"

That said, your question made me realize i overlooked something that should definitely be part of S5: We need to keep an eye out for issues that were backported to branch_6x after branch_6_0 was created, and were not backported to branch-6_0 – in which case fixVersion=6.0 needs removed in S5.

In my original brainstorming of this plan, i was going to do this audit twice (once before merging the versions when "master" could be removed from issues w/only "master" commits, and once after to add 6.0 in the paranoia case described above) making this extra check unneccessary. But this order seemed better because it means we only have to audit the list one time, w/o any time pressure and w/o the list continually growing as people resolve more 6.1 issues.

(S6 should also catch these issues, but i wanted redundency since it's possible people made issue# typos in the git commit messages)

I've updated the steps in the issue description accordingly.

@asfimport
Copy link
Author

Steven Rowe (@sarowe) (migrated from JIRA)

Looking at a different JIRA version problem (see http://markmail.org/message/6ac5szyce3qhoo3l), I noticed that LUCENE (and not SOLR) has version contstants 6.x and 5.x - there are 38 issues with these as fixVersion-s: https://issues.apache.org/jira/issues/?jql=project+in+(LUCENE,SOLR)+AND+(fixVersion=6.x+OR+fixVersion=5.x).

I think this is a related problem to the master->6.0 issue being dealt with here.

Should we even have 6.x and 5.x versions at all?

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

I think this is a related problem to the master->6.0 issue being dealt with here.

I think it is a similar class of problem, and a similar audit could/should be done to fix those issues, but the specific steps needed to assess those issues and deal with that confusion will be distinctly diff from what needs done here to fix "master" (which affects a few order of magnitude more issues)

I'd really prefer not discuss 5.x/6x anymore in this jira and confuse/complicate the issue anymore.

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

Step S1...

Attaching 2 files:

@asfimport
Copy link
Author

asfimport commented May 6, 2016 •
edited

Steven Rowe (@sarowe) (migrated from JIRA)

I'd really prefer not discuss 5.x/6x anymore in this jira and confuse/complicate the issue anymore.

I created #8330 for this.

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

Didn't even make it to Step #2 before i discovered a problem...

Jira's "Bulk Edit" feature has been seriously hobbled since the last time i used it...

  1. You can only bulk edit up to 1000 issues at a time now.
    • This makes it impossible to do a bulk edit on all 4K+ issues with fixVersion=master
    • I could work around this by crafting 5 distinct queries, but...
  2. The options you can do when Bulk Editing are really limited...
    • I tried doing a bulk edit on the 186 issues with fixVersion=6.0
    • the "Edit Issues" option was disabled with the text: NOTE: You do not have permission to edit the selected 186 issues or at least one issue has a status that forbids editing.
    • The remaining options are "Move Issues" (really bad idea), "Delete Issues" (even worse idea), "Transition Issues" (through workflow), "Watch Issues" and "Stop Watching Issues"
    • the start/stop watching issues options don't let you add a comment
    • the "Transition Issues" option seemed like a last hope, but it's too smart – it makes you pick a specific transition like "Resolved -> Closed" (no No-Op options listed) and knows that that each specific option will only affect a subset of the issues (the ones in the iniial state of the option you pick)

I think Step S2 needs to be scrapped. I would really like to have distinct comments identifying all of the issues, both because it would make it easy to search for all affected issues (and filter that search by other factors) but also so it shows up if/when people read the comments in these issues in case something gets missing in Steps S5 and S6 – but I just don't think that's possible.

My next best suggestion is to just run excel/csv reports for the queries in S2 (just like S1) and attache them to this issue for posterity.

I'll revise the description of the issue to reflect the new plan, and put everythine on hold until monday in case anyone has better suggestions.

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

revised description based on the fact that jira bulk edits are fucking useless now.

I'll start this process over again at S1 on monday unless i hear otherwise.

@asfimport
Copy link
Author

Cassandra Targett (@ctargett) (migrated from JIRA)

the "Edit Issues" option was disabled with the text: "NOTE: You do not have permission to edit the selected 186 issues or at least one issue has a status that forbids editing."

Oh right, if the issue is closed already, you can't edit the fix version at all (and it gives that rather unhelpful message about permissions). Assuming that is the problem for those issues, we would need to reopen all those in order to fix the version then close them again.

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

Assuming that is the problem for those issues, we would need to reopen all those in order to fix the version then close them

Bulk Edit is already stressfull enough, I'd really really rather not Bulk Edit once to re-open the subset of issues that are currently closed issues (and add one comment to keep track of them) then Bulk Edit (2 more times) to add the comments i originally wanted to add, then Bulk Edit a third time to re-close issues from the 1st time ...

...particulaly since we're only assuming that's the problem. who knows what other "workflow" constraints might be in place to prevent me from doing a No-op Bulk Edit to add a comment to all the issues (they've clearly added a lot more "smarts" to bulk editing lately, so i'm not even sure a No-Op edit that just adds a comment will even work at this point)

At this point I personally just really want to avoid the Bulk Edits – but i'm happy to hand this issue off to someone else if they feel more confident then me about being able to follow the original plan :)

@asfimport
Copy link
Author

Anshum Gupta (@anshumg) (migrated from JIRA)

Hoss, I need to close out the issues for 5.5.1 and also move the unresolved ones to 5.5.2 but I'm sure there's an intersection there with the issues here. If you haven't already started on that OR think that my update wouldn't interfere, I'd go ahead with mine.

  • change fixVersion from 5.5.1 -> 5.5.2 for all unresolved issues which also contain a fixVersion of 5.5.1 (might also contain master/6.0 here, which is what I'm worried about)
  • Closing out the issues with 5.5.1 as a fixVersion (again the same problem).

I'm traveling so I may only be able to check on this later at night (or right now).

@asfimport
Copy link
Author

Chris Hostetter (Unused) (migrated from JIRA)

Closing all issues with a certain fix version affects nothing.

You can't do any bulk changing of fix versions w/o completely replacing all fix versions on an issue, so don't bother trying - that's completely independent from anything I've got planned. Just leave unresolved issues alone, (or edit them individually)

-Hoss

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

Starting again today...

....AAAAANNNNNDDDDD Aparently Jira silently drops anything after the first 100 issues when exporting.

So, we can work around this by doing an export with "pagination", but for our 3 diff reports that's ~45 URLs to keep straight, so i'm going to try and bang out a little perl script to make sure i don't miss something.

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

Ok ... i will not let jira defeat me.

Attaching many files...

...moving on to S3 & S4

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

Versions merged ... no going back now.

  • S3
    • LUCENE: master merged into 6.0
    • SOLR: master merged into 6.0
  • S4
    • LUCENE: added master (7.0)
    • SOLR: added master (7.0)

Going ot grab some lunch and then start working my way through S5 and S6 this afternoon

FYI: Jira's "Merge Versions" feature is really weird (and really slow) now .... it spins for a while and then returns, but hte "old" version is still listed on the version screen (for a while). It appears that it spins up a background thread that iterates over every individual issue in the background and change them, causing concurrent queries you try from the browser for the "old" version to slowly see decreasing results over time, until evnetually there are no results found, and then the old version disappears from the list of Verions.

In fact: If you view "All" Activity for any of the issues that use to be assocaited with the (old) 'master' version, it says i edited the fixVersion and changed it to 6.0, and it says the "updated" date for all of these issues is "Just now"

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

Oh, and before i forget: Since it took so long to run the reports & merge the versions, I reviewed all jira email notifications from this morning (up until my last comment made it to my mail spool) to sanity check there were no issues that got "resolved" with fixVersion=master after/during running the above reports, but before i merged the versions – there was nothing there to worry about.

@asfimport
Copy link
Author

asfimport commented May 9, 2016 •
edited

Chris M. Hostetter (@hossman) (migrated from JIRA)

Status update...

  • S5
    • Making progress manually reviewing issues from the S1 report – about 50 issues left to review.
      • attaching LUCENE-7271_S5_hoss_todo.txt which is my personal checklist i'm working through (deleeting as i go)
  • S6
    • SOLR-4509 was the only issue in either CHANGES.txt 7.0 section, so i went ahead and updated it in jira to master (7.0)
    • Attaching LUCENE-7271_S6_report.txt containing the GIT SHAs on master but not 6.0.0 that still need reviewed
      • I won't bother worrying about this until Step # S5 is done, but I wanted to go ahead and generate this report now so the list of commits wouldn't keep growing with stuff i didn't need to worry about it.

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

S5 complete, moving on to S6.

@asfimport
Copy link
Author

Chris M. Hostetter (@hossman) (migrated from JIRA)

S6: DONE
LUCENE-7271: DONE

...and now returning to my regularly scheduled life.

@asfimport
Copy link
Author

Anshum Gupta (@anshumg) (migrated from JIRA)

Thanks Hoss!

This was referenced Aug 24, 2022
Open
Closed
Closed
Closed
Closed
Closed
This was referenced Aug 24, 2022
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hossman hossman

Labels
legacy-jira-priority:Major legacy-jira-resolution:Fixed type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hossman @asfimport