2021-07-07 planning discussion

[kgodey]

Updates

• I've made some updates to the order and deadlines of a few milestones for more efficient implementation.
  • 03. Tables for File Import has changed order from milestone 2 to milestone 3 and the deadline has been updated to July 19 to allow for additional time to complete backend work.
    • This means that 02. Basic Filtering, Sorting, Grouping has been moved to milestone 2. The deadline remains the same, July 12.
  • 04. Multiple Database Support has been moved from milestone 5 to milestone 4 after consultation with @pavish since it involves a major layout change which would be better to complete sooner rather than later.
    • This means that 05. Editable Tables is now milestone 5.
    • The deadlines for both have been swapped as well.
  • 11. User Management has been moved from milestone 16 to milestone 11 because having user accounts sooner rather than later will help define future features.
    • We've already run into issues where we can't implement issues like Favorites because of the lack of user accounts.
    • Due dates of future milestones have been adjusted to compensate.
  • Added 26. Post-MVP milestone to track issues we plan to get done after the MVP.
• @eito-fis: Please note that I've assigned a bunch more issues to you since you seemed close to being finished with what you have so far. They should all be pretty small.
• I added a new "Hiring" section in upcoming work since I'll be doing some work on that.

Questions

• How's the asynchronous planning going so far?
  • Any concerns?
  • Anything you think would have gone faster if we'd been synchronous?
• Does your workload for the week look manageable? Too much/too little?
• Any comments or questions about the retrospective?
• Any comments or questions about the milestone updates?
• Milestone 02. Basic Filtering, Sorting, Grouping is due on Monday. Any blockers?
• Anything else you'd like to add?

Retrospective

This covers work since our last discussion on 2021-06-30. This was a shorter week due to the CCI holiday for U.S. Independence Day.

Product

• Rearranged milestones (see the Updates section above) to reflect better implementation order.
• Re-reviewed milestones 2-7 and created some new issues to cover additional work.
  • 04. Multiple Database Support
    • User should be able to configure multiple databases in settings
    • Create new /v0/databases/ API
    • Switch frontend code to the database API
    • Remove database keys API
    • Only one schema should be open at a time
    • Ensure that only Postgres databases are connected
  • 07. Working with Tables
    • Design for editing column properties and table constraints
  • 26. Post-MVP
    • Support for handling hundreds of DBs/tables/schemas
    • CRUD API for table constraints
    • Frontend work for editing column properties
    • Frontend work for working with table constraints
• Created a bunch of issues to add components to our Figma component library
  • Button
  • Input
  • Tab
  • Sidebar
  • Collapsible header
  • Table

Design

• Design wiki has been updated to have better organization and be friendlier to new contributors
• Finalized design specs and feedback for milestone 4. Multiple Database Support
• Finalized design for updated components for 2. Tables for File Import milestone
• Started design for 6. Working with Schemas

Development

• Backend work on 02. Basic Filtering, Sorting, Grouping
  • Record filtering backend and API completed
  • Record sorting backend and API completed
  • Record grouping backend and API completed
  • sqlalchemy-filters fork that works with 1.4 style queries created
  • Test DB handling issue raised
• 03. Tables for File Import milestone
  • Frontend work:
    • Setup storybook.
    • Documentation and Tests for components: Button, TextInput, Icon and Skeleton.
    • Started work on:
      • File upload component
      • Switch to new CSV API
      • File import page
  • Backend work:
    • Column alteration API completed
    • Spec for confirm data APIs completed: Import Preview API
• Backend work on 04. Multiple Database Support
  • Support configuration for multiple DBs complete
  • Tests for multiple dbs in progress
  • Issue with DB handling in tests fixed

Infrastructure

• Continued work on Ansible setup for staging server.
  • Databases are now set up and running.

Active Work

This is work planned for the upcoming week.

Product

• @kgodey: Review milestone 08. Initial Data Types and make/modify issues as needed.
• @kgodey: Review newly created component library issues and sort them into milestones.

Design

• @ghislaineguerin: Complete design feeback and specs for 6. Working with Schemas
• @ghislaineguerin: Start design for multiple design issues in 7. Working with Tables
• @ghislaineguerin: Start design for 8. Initial Data Types
  • Start with text and number types.
• @Dhruvi16: Create button component in the component library

Development

• 02. Basic Filtering, Sorting, Grouping
  • @pavish: Break down frontend ticket Implement filtering, sorting, and grouping in the UI into smaller issues and start work on them.
• 03. Tables from File Import
  • Frontend
    • @pavish: Expecting to complete these in first couple days of week.
      • File upload component
      • Switch to new CSV API
      • File import page
    • @pavish: File import preview page, could be delayed based on backend work completion for confirm data APIs.
  • Backend
    • @eito-fis: Remove legacy CSV functions once frontend no longer uses them.
    • @mathemancer: Data confirmation step API work
    • @mathemancer: engine caching fix to avoid over-creation of connections
• 04. Multiple Database Support
  • @eito-fis: Ensure only Postgres databases are connected.
  • @eito-fis: Create databases API that show supported types.
  • @eito-fis: Test imports from multiple databases.
• 06. Working with Schemas
  • @eito-fis: Deleting and renaming schemas
  • @eito-fis: API for deleting and renaming schemas
• 07. Working with Tables
  • @eito-fis: Deleting and renaming tables
  • @eito-fis: API for deleting and renaming tables
  • @eito-fis: Allow users to create empty table via API
  • @eito-fis: API to upload file from URL

Infrastructure

• @kgodey: Continue setup of staging server
• @kgodey: Deploy Storybook (if time permits)

Hiring

• @kgodey: Hiring plan and conversations for new frontend engineer and backend engineer support for @mathemancer's leave.

[mathemancer]

• How's the asynchronous planning going so far?
  I think it's going swimmingly so far.
  • Any concerns?
    None at the moment.
  • Anything you think would have gone faster if we'd been synchronous?
    PR turnaround is always tricky, but I think it's well within a reasonable timeframe in most cases.
• Does your workload for the week look manageable? Too much/too little?
  I think it's reasonable.
• Any comments or questions about the retrospective?
  None.
• Any comments or questions about the milestone updates?
  None.
• Milestone 02. Basic Filtering, Sorting, Grouping is due on Monday. Any blockers?
  Nothing that I see.
• Anything else you'd like to add?
  I think we should consider adding Mapping DB data types to Mathesar data types #269 to Milestone 3 -- Tables from File Import, since otherwise the way we do type previewing will be kind of icky. This is because the current representation when viewing a table of the type of a column is the actual underlying DB type. But, the required type submission when previewing an alteration (or patching the table) is actually the service-layer name for the type. Syncing these representations was part of Mapping DB data types to Mathesar data types #269 (I think).

[kgodey]

@mathemancer I fleshed out #269 a bit more. I don't think we can add it to 03. Tables from File Import because it depends on the database API from 04. Multiple Database Support being completed.

I see three options:
(1) We swap milestones 3 and 4
(2) We implement it messily and refactor later
(3) We use DB layer types or service layer types everywhere for now until we get to #269. It should be possible to use service layer types in the column/table API.

What do you think?

[mathemancer]

For now, how about I just try to take the shortest route to having the same string representation of a type being shown when viewing tables and columns as needs to be submitted when creating / altering columns?

Edit: Shortest route, subject to being reusable when we want to change to something more human-friendly.

[kgodey]

Sounds good.

[ghislaineguerin]

How's the asynchronous planning going so far?

Any concerns?

No concerns.

Anything you think would have gone faster if we'd been synchronous?

I think reviewing the milestones and asking product definition questions before implementing is going to help increase velocity.

Does your workload for the week look manageable? Too much/too little?

It is manageable.

Any comments or questions about the retrospective?

No questions.

Any comments or questions about the milestone updates?

No questions.

Milestone 02. Basic Filtering, Sorting, Grouping is due on Monday. Any blockers?

None

Anything else you'd like to add?

I think we've covered this during our call today regarding moving design to review much faster so that the review feedback can be used to improve the current design work and avoid inconsistencies and rework.

[kgodey]

@ghislaineguerin Sounds good, thanks. I'll leave it up to you to let me know when calls or questions would be helpful to increase velocity, let me know if I can help in any other way.

[pavish]

• How's the asynchronous planning going so far?
  I think it's going pretty good.

  • Any concerns?
    None, so far.
  • Anything you think would have gone faster if we'd been synchronous?
    I don't see any such scenario. I think it's going good.

• Does your workload for the week look manageable? Too much/too little?
  It's a bit tightly packed for this week on the frontend. I've not yet created the necessary issues for Milestone #2.

• Any comments or questions about the retrospective?
  None.

• Any comments or questions about the milestone updates?
  None.

• Milestone 02. Basic Filtering, Sorting, Grouping is due on Monday. Any blockers?
  Has a lot of work on the frontend, I've not begun on them yet. Also, I'm planning on moving to a different city this weekend, I might not have enough time on Monday. This milestone will get delayed.

• Anything else you'd like to add?
  I've improvised some design during development, like showing progress bar for file upload, error notification component etc., I'll schedule a call with @ghislaineguerin to show how the current UI looks like. The staging env would be more helpful for everyone to be upto date.

[kgodey]

@pavish Would moving the milestone by 3 days give you sufficient time?

[pavish]

@kgodey Yes. 3 days would be sufficient time, with additional buffer period.

[kgodey]

@pavish I've updated the deadline to July 15.

[kgodey]

How's the asynchronous planning going so far? Any concerns? Anything you think would have gone faster if we'd been synchronous?

I have no concerns, I think it's going great. It takes me 45 min or so to put the weekly planning thread together each week, I'll ask a question about how useful they are and what parts are useful next week to make sure I'm spending my time on the right things.

Does your workload for the week look manageable? Too much/too little?

It's fine, I hope to make quicker progress on the staging server.

Any comments or questions about the retrospective?
Any comments or questions about the milestone updates?

No.

Milestone 02. Basic Filtering, Sorting, Grouping is due on Monday. Any blockers?

It looks like there's too much work there for this week on the frontend, so I'll move the deadline.

Anything else you'd like to add?

No.

[eito-fis]

How's the asynchronous planning going so far?

Good - feel pretty comfortable with this working style at this point.

Does your workload for the week look manageable? Too much/too little?

Decent amount, but I think a fair bit of the tasks are relatively small so I think I'll be alright.

Any comments or questions about the retrospective?
Any comments or questions about the milestone updates?
Milestone 02. Basic Filtering, Sorting, Grouping is due on Monday. Any blockers?

No.

[Dhruvi16]

- How's the asynchronous planning going so far?
I am getting familiar with the milestones and to me, it looks great.

 - Any concerns?
 Nope.
 - Anything you think would have gone faster if we'd been synchronous?
 I am not sure

- Does your workload for the week look manageable? Too much/too little?
It is manageable for me.
- Any comments or questions about the retrospective?
Nope
- Any comments or questions about the milestone updates?
Nope
- Milestone 02. Basic Filtering, Sorting, Grouping is due on Monday. Any blockers?
Nope
- Anything else you'd like to add?
Nope