2021-09-07 weekly planning

[kgodey]

Discussion topics

• Updates: General updates everyone should be aware of.
• Questions: Questions for the team to answer.
• Active Work: Work planned for the next week.
• Retrospective: Summary of last week's work.
• Editable Tables & deadline updates

About

Each Tuesday, we start a discussion about Mathesar's progress. Everyone is welcome to add a new topic to discuss or to comment on existing topics.

[kgodey]

Updates

Progress

• 05. Editable Tables is due on Friday.
• 06. 2021-09 Stability is now mostly fleshed out and we're making progress on backend stability work with two major reorganization pull requests merged this week.
• 07. Working with Tables has only frontend issues remaining and the first of those was completed this week.
• 08. Initial Data Types is also moving along with both frontend and backend issues in progress.
• 09. Working with Views is in the design stage and three design specs are in the review process.

State of Mathesar meetings

The core team had two long "State of Mathesar" meetings in the last week where we reviewed the progress we've made so far and how we've been working together. This has resulted in several changes to the roadmap and processes. You can see the full details in the meeting notes on our wiki, but highlights include:

• The alpha release roadmap has been updated to remove high-cost features including visualizations, product-wide search, real time collaboration, data export, improved spreadsheet-style editing, undo & redo for data modeling, and finalization of our API.
• We've added a couple of things to the alpha release plan, including public sharing of views and forms, a better editing experience, and time for user testing and launch preparation.
• We've created a plan and timeline for a beta (public) release.
• We've updated our standup format, GitHub workflow, and design review process.

Contributors

• We have one new contributor with a merged pull request, @seancolsen!
• @powellc had his first PR merged in, closely followed by a second one.
• @linzhou-zhong and @mr-gabe49 continued to have new PRs merged in.

Thanks for your work. :)

[kgodey]

Questions

@mathemancer, @powellc, @ghislaineguerin, @pavish: Your responses are required. Others are also welcome to respond.

Retrospective

• What went well over the past week?
• What could have gone better over the past week?
• Are there any improvements you'd like to suggest?
• If you attended the State of Mathesar meetings, was it useful? What did you get out of it?

Active Work

• Does your workload for this week look okay?
• We have one milestone due in the next week. Are there any blockers to meeting the deadline?

Discussion

Please respond to the Updates post directly if you'd like to discuss anything there. If you have any other topics to discuss, please post them as new replies to the parent post.

[ghislaineguerin]

What went well over the past week?

I think the shorter deadline for reviews is working well, feedback has been timely and helped me inform the design as I was working on the next issue.

What could have gone better over the past week?

I think overall it was good. Can't think of any specific improvement, other than to be more mindful of checking links and content of things I'm posting to catch any issues.

Are there any improvements you'd like to suggest?

None at the time.

If you attended the State of Mathesar meetings, was it useful? What did you get out of it?

It was helpful to align as a team. It was great to hear feedback regarding the design process as well.

Does your workload for this week look okay?

It does

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

None

[mathemancer]

What went well over the past week?

Getting back into the swing of things. It's good to catch up with the team.

What could have gone better over the past week?

I was considerably less productive than I'd hoped over the last week. I also realized partway through my work that there was much more to be done to implement the MATHESAR_TYPES.MONEY type than had been needed for the MATHESAR_TYPES.EMAIL type.

Are there any improvements you'd like to suggest?

Nothing that we didn't come up with during the State of Mathesar meetings.

If you attended the State of Mathesar meetings, was it useful? What did you get out of it?

I think they were useful. Objectively, we ended up with a list of action items that I don't think we would have arrived at any other way.

Does your workload for this week look okay?

It's a lot, but I'll start at the top and work my way through things.

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

None that I see.

[pavish]

What went well over the past week?

I started on some refactors I've been wanting to do for a while. I've also been sticking to personal schedules, only focusing on the tasks I've mentioned on the standup, each day.

What could have gone better over the past week?

I underestimated the effect of refactoring the stores. It touches a lot of areas, and not having tests for the frontend is slowing me down. All the more reason to write tests with maximum coverage, in the stability milestone.

Are there any improvements you'd like to suggest?

No. I like the dependencies section in our standups, works great for me for reviewing designs.

If you attended the State of Mathesar meetings, was it useful? What did you get out of it?

I found it very useful. It gave an overall view/clarity on our current state. I am hoping we don't have to cut down more features, but based on the UI progress, I guess we may have to.

Does your workload for this week look okay?
We have one milestone due in the next week. Are there any blockers to meeting the deadline?

It will probably take this week to complete the refactoring. In the meanwhile, I do not want to merge other frontend PRs, since this changes some fundamental approaches.

Editing and adding new rows should not take a lot of time. I'm estimating 2 to 4 days for both combined. It would be better if we move the milestone's deadline to next week, 15th.

[kgodey]

What went well over the past week?

I'm glad to be refactoring the db code, it will be easier to work with in the future and hopefully more approachable to new contributors. I also think the State of Mathesar meetings went well, it was good to catch up with the team and get on the same page.

What could have gone better over the past week?

The db code reorganization is taking me longer than I thought.

Are there any improvements you'd like to suggest?

I'm trying out starting a new topic in this discussion to branch off of questions answered here.

If you attended the State of Mathesar meetings, was it useful? What did you get out of it?

It was useful for me. I think our roadmap is more cohesive and we had a chance to reflect on how we work together and how to improve it.

Does your workload for this week look okay?

I'm probably not going to be able to finish everything because of the db code organization taking longer, but that's okay.

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

Addressed in #644 (comment)

[powellc]

What went well over the past week?

Not a lot. Having Monday off threw me off to start the week and then a number of things outside of work conspired to make this a really frustrating week for getting things done. Will look to open a PR for more testing around API column alterations tomorrow and get a start on adding numeric ranges. 🤞🏽

What could have gone better over the past week?

Lost a lot of focus with the short week due to the US holiday and non-work issues. Hoping to be productive tomorrow.

Are there any improvements you'd like to suggest?

Not at the moment

If you attended the State of Mathesar meetings, was it useful? What did you get out of it?

N/A ... ended up having other meetings booked over the top :(

Does your workload for this week look okay?

Given my lack of focus, no, but I'll do my best to dig out with the last day of the week

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

I don't see any

[kgodey]

Active Work

This post lists work planned for this week (see also: Active). It also includes a current summary of milestones involved in this week's work.

[2021-09-10] 05. Editable Tables

• 8 issues already closed.
• Issues currently in progress:
  • [@pavish] Implement record editing #195
• Issues not yet started:
  • [@pavish] Implement adding new records #197

[2021-09-17] 06. 2021-09 Stability

• 2 issues already closed.
• Issues currently in progress:
  • [@kgodey] Reorganize code in db and set some standards for imports #582
  • [@andyharley-cci] Set up staging environment for Mathesar #225
• Issues not yet started:
  • [@kgodey] Refactor db code to work with existing connections #592
  • [@kgodey] Schema sqlalchemy engine caching is per-instance #280
  • [@kgodey] Handle validation, errors, and business logic in the API consistently #623
  • [@kgodey] Set up logging in the backend code #583
  • [@ghislaineguerin] General styling updates #473
  • [@ghislaineguerin] Design for loading, error, and retry states #636
  • [@ghislaineguerin] Review existing UI and create tickets for changes needed #638
  • [@pavish] Set up integration testing environment #639
  • [@pavish] Write integration tests for some important user scenarios #640
  • [@pavish] Unit tests for stores #641
  • [@pavish] Unit tests for all reusable components #642
  • [@pavish] Move filter pane to dropdown #643
  • [@pavish] Frontend unit test cases: Readonly table view #213
• Issues remaining in backlog:
  • 4 backend and database issues
  • 2 frontend issues

[2021-09-23] 07. Working with Tables

• 20 issues already closed.
• Issues currently in progress:
  • [@dmos62] Implement showing and changing a column's type #498
  • [@pegnott] Implement adding a new column #495
• Issues not yet started:
  • [@mr-gabe49] Implement deleting a table #448
• Issues remaining in backlog:
  • 11 frontend issues

[2021-09-27] 08. Initial Data Types

• 27 issues already closed.
• Issues currently in progress:
  • [@mathemancer] Backend work for Mathesar money custom type  #417
  • [@eito-fis] Handle TIME data type in the backend #426
  • [@powellc] Implement filtering options for Number types. #385
  • [@powellc] Handle BOOLEAN data type in the backend #387
  • [@kgodey] Spec for implementing display options for Columns #392
• Issues not yet started:
  • [@kgodey] Update implementation issues for Initial Data Types milestone #552
  • [@powellc] Implement range based grouping for numeric types. #384
  • [@mathemancer] Create URI custom type and handle it in the backend. #412
  • [@mathemancer] Handle TIMESTAMP data type in the backend #424
  • [@mathemancer] Handle INTERVAL type in the backend #430
  • [@mathemancer] Handle MONEY data type in the backend. #420
  • [@powellc] Implement filtering options for the email type #410
  • [@mathemancer] Handle VARCHAR data type in the backend. #404
  • [@mathemancer] Handle TEXT data type in the backend. #405
  • [@mathemancer] Handle CHAR data type in the backend. #407
• Issues remaining in backlog:
  • 19 backend and database issues

[2021-09-30] 2021-09 improvements

• 5 issues already closed.
• Issues currently in progress:
  • [@asharonbaltazar] CellHeader dropdown floats above the sidebar #594
  • [@arvind-1] Column API should support creating columns without names  #555
  • [@andyharley-cci] Handle non-unicode encoded datafiles #421
  • [@mr-gabe49] Select component: When scrollbar is present within the dropdown, active option should always be visible #608
• Issues remaining in backlog:
  • 10 frontend issues
  • 3 infrastructure issues

[2021-10-04] 09. Working with Views

• 4 issues already closed.
• Issues currently in progress:
  • [@ghislaineguerin] Support for editing records within a view #456
  • [@ghislaineguerin] Support for opening and interacting with a View #466
  • [@ghislaineguerin] Support for deleting a database view #454
  • [@ghislaineguerin] Design for usage of Foreign Key constraints #243
• Issues not yet started:
  • [@ghislaineguerin] Support for listing database views #455
  • [@ghislaineguerin] Design for many-to-many relationships between tables  #443
• Issues remaining in backlog:
  • 7 backend and database issues
  • 1 other issues

[2021-10-18] 11. Multiple Database Support

• 7 issues already closed.
• Issues currently in progress:
  • [@aaditkamat] Backend support for tables without single-column primary keys #395
• Issues remaining in backlog:
  • 3 backend and database issues
  • 3 frontend issues
  • 2 design issues

[kgodey]

Retrospective

This post lists issue updates from 2021-08-31 until now by milestone.

2021-08 improvements

• Issues closed:
  • [@kgodey] Serve built static files in production #358

05. Editable Tables

• Issues started, not yet finished:
  • [@pavish] Implement record editing #195

06. 2021-09 Stability

• Issues closed:
  • [@pavish] Permissions Issues Within Docker Containers #626
  • [@kgodey] Split up API and serializer code by object #510
• Issues started, not yet finished:
  • [@kgodey] Reorganize code in db and set some standards for imports #582
  • [@andyharley-cci] Set up staging environment for Mathesar #225
• New issues created:
  • Move filter pane to dropdown #643, created by @kgodey
  • Unit tests for all reusable components #642, created by @kgodey
  • Unit tests for stores #641, created by @kgodey
  • Write integration tests for some important user scenarios #640, created by @kgodey
  • Set up integration testing environment #639, created by @kgodey
  • Review existing UI and create tickets for changes needed #638, created by @kgodey
  • Implement design for loading, error, and retry states  #637, created by @kgodey
  • Design for loading, error, and retry states #636, created by @kgodey
  • Table, Database, & Schema APIs should support sorting by ID & name #635, created by @kgodey
  • Ensure the Tables list is sorted after adding a new empty table #631, created by @seancolsen
  • Permissions Issues Within Docker Containers #626, created by @eito-fis
  • Handle validation, errors, and business logic in the API consistently #623, created by @kgodey
  • Enable TypeScript strict mode #619, created by @pavish

07. Working with Tables

• Issues closed:
  • [@seancolsen] Implement adding a new empty table #444
• Issues started, not yet finished:
  • [@dmos62] Implement showing and changing a column's type #498
  • [@pegnott] Implement adding a new column #495

08. Initial Data Types

• Issues closed:
  • [@powellc] Implement filtering options for Boolean types #388
  • [@ghislaineguerin] Design for duration data type #241
  • [@ghislaineguerin] Design for date & time data types #240
• Issues started, not yet finished:
  • [@mathemancer] Backend work for Mathesar money custom type  #417
  • [@eito-fis] Handle TIME data type in the backend #426
  • [@powellc] Implement filtering options for Number types. #385
  • [@powellc] Handle BOOLEAN data type in the backend #387
  • [@kgodey] Spec for implementing display options for Columns #392

2021-09 improvements

• Issues closed:
  • [@mr-gabe49] UpArrow does not work in the group section dropdown when data is grouped. #609
  • [@linzhou-zhong] Uncaught 404s in record viewsets and pagination classes #517
  • [@mr-gabe49] Changes to sorting order behavior #474
  • [@linzhou-zhong] Set up GitHub Action to triage issues automatically when an issue has a status #270
• Issues started, not yet finished:
  • [@asharonbaltazar] CellHeader dropdown floats above the sidebar #594
  • [@arvind-1] Column API should support creating columns without names  #555
  • [@andyharley-cci] Handle non-unicode encoded datafiles #421
  • [@mr-gabe49] Select component: When scrollbar is present within the dropdown, active option should always be visible #608
• New issues created:
  • Preserve table scroll across live table updates #632, created by @dmos62

09. Working with Views

• Issues started, not yet finished:
  • [@ghislaineguerin] Support for editing records within a view #456
  • [@ghislaineguerin] Support for opening and interacting with a View #466
  • [@ghislaineguerin] Support for deleting a database view #454
  • [@ghislaineguerin] Design for usage of Foreign Key constraints #243

11. Multiple Database Support

• Issues started, not yet finished:
  • [@aaditkamat] Backend support for tables without single-column primary keys #395

[kgodey]

Editable Tables & deadline updates

Starting a new topic to discuss @pavish's reply here: #644 (reply in thread)

It will probably take this week to complete the refactoring. In the meanwhile, I do not want to merge other frontend PRs, since this changes some fundamental approaches.

Editing and adding new rows should not take a lot of time. I'm estimating 2 to 4 days for both combined. It would be better if we move the milestone's deadline to next week, 15th.

• I've updated the "Editable Tables" milestone deadlines to be the 15th.
• I've also updated the next few milestones so that the deadlines are all clustered towards the end of September and early October. We should be aiming to finish these much earlier, but then I don't have to track/move them on a day-by-day basis. Please see the roadmap

@pavish can you create a new issue to track refactoring work and put it in 06. 2021-09 Stability? Also, would you be able to get that merged in before editing/adding so that other frontend PRs are unblocked?

[pavish]

@kgodey I've created #651, and placed it in 06. 2021-09 Stability.

I can get this merged in before editing/adding.