General Toolset Improvements (WIP)

[nbrady-techempower]

General Toolset Improvements (Work in progress)

Motivations

Toolset Roadmap

There are a few things I'd like to accomplish here. First, I'd like to start future-proofing the toolset for a Python upgrade. We may not be far off from being able to move to 16.04, and even 18.04 when stable is released in April. This will ship with python 3.6 as the default. In the meantime we can do a few things to make the toolset work in 16.04/18.04/3.6 and still be compatible with 14.04/python 2.7.

I'd also like to explore getting some test coverage for the toolset. Ideally this would be tests before the matrix of framework tests that would only run if a toolset file has been modified.

Modularize benchmarker.py and framework_test.py into a subdirectory of more manageable pieces. This doesn't mean a complete rewrite, but I'd like to take a closer look at these files and see if we can better document/reuse come components to make the toolset more approachable. We rarely, if ever, get help with toolset improvements. This could also be a good time to enforce some code style consistency with PEP.

• Replace python print statements with function calls
• Audit integer division (python 3 now converts to float)
• Look into __future__ import for compatibility with helpful python 3 features
• caniusepython3 or some similar tool to check modules for compatibility issues
• Include important work from #2536 - More robust verification - I got you @knewmanTE
• Create at least one test for the toolset and add to travis
• Create a structure in /toolset to modularize and group related code
• Look into #3188 - bubbling up bash toolset errors
• Use a 16.04 docker image in a separate travis matrix to test a few known working frameworks to see what's blocking a path to upgrade
• Examine fortune parser for more helpful error messages
• Remove _id from mongo setup and make sure all frameworks using mongo use the id field which is now properly indexed (see #3242)

Note: I may cherry pick from this PR if I can fix some of the issues listed before the overall improvemnts are done

[knewmanTE]

Not sure how we feel about automated code migration, but there is a tool called 2to3-3.6 that automatically updates Python 2 files.

Due to the nature of TFB and the importance of not breaking existing code, I can understand why it may be preferable to manually make the changes, but I figured it was worth mentioning.

[knewmanTE]

Not sure how we feel about automated code migration, but there is a tool called 2to3-3.6 that automatically updates Python 2 files.

Due to the nature of TFB and the importance of not breaking existing code, I can understand why it may be preferable to manually make the changes, but I figured it was worth mentioning.

[nbrady-techempower]

@knewmanTE Yeah I saw that recently when I was looking at the caniusepython3 thing. I think we're actually pretty close as it is.

Plus at this point, the update verifications that you started is essentially done. I have a few more frameworks that I have to look at, but I'll probably look to have this merged once that's complete, as I think that update verification is super important.

[nbrady-techempower]

@knewmanTE Yeah I saw that recently when I was looking at the caniusepython3 thing. I think we're actually pretty close as it is.

Plus at this point, the update verifications that you started is essentially done. I have a few more frameworks that I have to look at, but I'll probably look to have this merged once that's complete, as I think that update verification is super important.

[knewmanTE]

@nbrady-techempower that's very exciting to hear! Thanks for looking into the update verification stuff. It was frustrating knowing that it was so close but there were still those few random frameworks that were breaking, but I was always holding onto the hope that if we waited a bit for some general TFB cleanup/updates/improvements, some of the issues would resolve on their own.

[knewmanTE]

@nbrady-techempower that's very exciting to hear! Thanks for looking into the update verification stuff. It was frustrating knowing that it was so close but there were still those few random frameworks that were breaking, but I was always holding onto the hope that if we waited a bit for some general TFB cleanup/updates/improvements, some of the issues would resolve on their own.

[nbrady-techempower]

So we're going to merge this in now mainly for the Update Test Verification. There are no breaking changes here, except that some frameworks were getting by without committing update writes to the database. Those frameworks are now correctly marked as failing.

[nbrady-techempower]

So we're going to merge this in now mainly for the Update Test Verification. There are no breaking changes here, except that some frameworks were getting by without committing update writes to the database. Those frameworks are now correctly marked as failing.