2021-07-14 planning discussion

[kgodey]

Updates

Table pagination update

We're currently paginating all tables (using limit/offset pagination) instead of using infinite scroll. Our eventual plan has been to have an infinitely scrollable spreadsheet UI, but we settled with pagination to get the MVP done quickly. However, the rest of our UX is still based on a spreadsheet-like experience.

White working on grouping, Pavish realized that in order to support infinite scroll in the future, we'll end up having to redesign and rewrite the fundamental structure of our frontend code. In order to remove that overhead, it's better to take the time to implement infinite scroll now, which we've already started. On the frontend, we now have a working virtual scroll component ported from React (react-window) to Svelte.

Given this, we should switch to using infinite scroll in our design for up to 25k rows. If the number of rows are higher than 25k, we should have a combination of virtual scroll + pagination (which is not an immediate requirement for the MVP).

This will require some updates to our current plan:

• I don't think we need to update previous designs, we can use the existing design for pagination past 25k rows and if @pavish runs into any issues with implementing previous design specs (e.g. editing tables), we can solve those as we go along.
• Future designs should assume infinite scroll until 25k rows.
• We'll need to spec out how to handle tables without a primary key or with multi-column primary keys (this needs to happen regardless of infinite scroll).
• We'll need to ensure that the API denotes which column is the primary key to support merging rows by primary key
• We may need to sort rows by some field by default to make the experience of scrolling and updating rows at the same time smoother
• We're still going to be deferring considering cursor based navigation to post-MVP, but I'll create an issue to track it.

I'll spec out the issues that are necessary here.

Milestone updates

• Deadline for 02. Basic Filtering, Sorting, Grouping has been extended to July 19. Reasoning:
  • Virtual scroll implementation will take an extra couple of days. This was implemented to make grouping easier. It's important to do this soon because it affects the fundamental table structure.
• 05. Tables from File Import has moved from milestone 3 from milestone 5. Reasoning:
  • Backend work is still pending and is more complex than the work for the other milestones.
  • 04. Working with Schemas involves major layout changes on the frontend which would be better to complete sooner.
• 10. Multiple Database Support has jumped from milestone 4 to milestone 10. Reasoning:
  • The design for multiple databases has been obsoleted by layout changes in 04. Working with Schemas, so design for database switching will need to be re-done.
  • It makes more sense to focus on the design for multiple databases after we've worked out how to work well within a single database. Views, data modeling, etc. should be figured out first since they will also affect navigation and layout.

Questions

• How did this week go?
  • What went well?
  • What could have gone better?
• Feedback on weekly update process:
  • Is this weekly update useful for you?
  • Do you read everyone else's responses to the discussion or just the original post?
  • What portions of the weekly update do you find most/least useful?
  • Can you think of any improvements to the process of collecting data for the weekly update?
  • Can you think of any improvements to other aspects of the weekly update process?
• 02. Basic Filtering, Sorting, Grouping is due this week. @pavish, you're the main person with assigned issues. Any blockers?
• Please re-review the Roadmap. Does the order of the current milestones and their associated deadlines make sense?
• Feedback on the infinite scrolling:
  • Does the plan above make sense? Do you have any concerns?
  • Are we missing any work that needs to be specced out or designed?

Retrospective

This covers work since our last discussion on 2021-07-07.

Product

• Milestone updates (see above)
• Reopened multiple databases design issue since original design has been superseded
• Created issues related to staging server

Design

• 02. Basic Filtering, Sorting, Grouping
  • Started work on the button component in the Figma component library.
• 04. Working with Schemas
  • Finalized design spec and feedback for 'Working with Schemas'
• 05. Tables from File Import
  • Finalized design spec and feedback for 'Import Status' component
• 06. Working with Tables
  • Started design review for 'Basic Table Operations'
  • Started design for 'Editing column properties and table constraints'

Development

• 02. Basic Filtering, Sorting, Grouping
  • Backend
    • Fix for filtering count value
    • For for grouping count values
    • Fix for IndexError when opening Mathesar without any tables
  • Frontend
    • Started work on sorting and grouping
• 04. Working with Schemas
  • Backend
    • Renaming and deleting schemas in progress
• 05. Tables from File Import
  • Backend
    • Removed legacy CSV functions
    • Ongoing work on APIs for data checking import flow
  • Frontend
    • File import API switch
    • Tab state converted to dedicated store
    • Components:
      • File upload drag and drop component
      • Progress
      • Notification
• 06. Working with Tables
  • Backend
    • Renaming and deleting tables in progress
• 10. Multiple Database Support
  • Backend
    • Multiple database import tests
    • Ensure only Postgres databases are connected
    • Database endpoint in progress

Infrastructure

• Not much progress on the staging server this week.

Hiring

• Frontend hiring plan is almost complete
• Started reaching out to people for temporary backend contract

Active Work

This is work planned for the upcoming week.

Product

• @kgodey: Break up data types issues into smaller issues
• @kgodey: Create issues to modularize installation of Mathesar
• @kgodey: Create issues for infinite scrolling implementation
• @kgodey: Spec out milestone 9 (Working with Views)

Design

• 02. Basic Filtering, Sorting, Grouping
  • @Dhruvi16: Continue work on the button component in the Figma component library.
• 03. Editable Tables
  • @Dhruvi16: Work on the input component in the Figma component library
  • @Dhruvi16: Work on the tabs component in the Figma component library
• 06. Working with Tables
  • @ghislaineguerin: Finalize review for 'Basic Table Operations'
  • @ghislaineguerin: Finalize design for 'Editing column properties and table constraints'
• 07. Initial Data Types
  • @ghislaineguerin: Start design for 'Text Data Type'
  • @ghislaineguerin: Start design for 'Number Data Type'
  • @ghislaineguerin: Start design for 'Money Data Type'
  • @ghislaineguerin: Start design for 'Date & Time Data Type'

Development

• 02. Basic Filtering, Sorting, Grouping
  • Frontend
    • @pavish: Complete sorting and grouping
    • @pavish: Complete filtering
    • Pagination
      • @pavish: Virtual scroll component
      • @pavish: Update existing paginated table to a spreadsheet layout
    • @patelaryan7751: Improve code around frontend utilities
• 04. Working with Schemas
  • Backend
    • @eito-fis: Renaming and deleting schemas
    • @eito-fis: API for renaming and deleting schemas
• 05. Tables from file import
  • Backend
    • @mathemancer: Ongoing work on data checking API flow
    • @mathemancer: bringing coherence to SQLAlchemy engine usage
• 06. Working with Tables
  • Backend
    • @eito-fis: Renaming and deleting tables in progress
    • @eito-fis: API for renaming and deleting tables
    • @eito-fis: Allow user to create a new table via pasting from spreadsheet
    • @eito-fis: CRUD API for table constraints
    • @kgodey: Allow users to create empty table via API
    • @kgodey: Allow users to create data file via file upload
• 10. Multiple Database Support
  • Backend
    • @eito-fis: Database endpoint

Infrastructure

• @pavish: Set up frontend build for staging server
• @andyharley-cci: Set up Mathesar staging server
• @kgodey: Set up API docs
• @dhruvkb: Document pre-commit linter hook

Hiring

• @kgodey: Start reaching out to potential frontend candidates
• @kgodey: Create job description and post frontend job
• @kgodey: Continue reaching out to backend candidates

[ghislaineguerin]

How did this week go?
What went well?

Simplifying the design spec process and having discussions in one place is still having an impact on getting to the review stage faster.
Discussion w/ Pavish on front-end requirements helped to solve some design issues with navigation.
Having synchronous and async product discussions w Kriti seems to help w clearing up doubts about the design and scoping better.

What could have gone better?

Could have solved database switching better if navigation had been handled globally rather than focusing on the schema view.

Feedback on weekly update process:
Is this weekly update useful for you?

Yes.

Do you read everyone else's responses to the discussion or just the original post?

I do, yes.

What portions of the weekly update do you find most/least useful?

I think planning the active work helps to set priorities, plan ahead to complete the work.

Can you think of any improvements to the process of collecting data for the weekly update?

I think it's working fine, can't think of any improvements at the moment. We might try asking clarifying questions to other posts in order to dig further into what they are doing.

Can you think of any improvements to other aspects of the weekly update process?

Not right now.

Please re-review the Roadmap. Does the order of the current milestones and their associated deadlines make sense?
Yes, they do.

Feedback on the infinite scrolling:
Does the plan above make sense? Do you have any concerns?

No concerns, I think it's a good decision. I'm wondering if we need to revisit the first milestone for read-only tables.

Are we missing any work that needs to be specced out or designed?

See above re: read-only tables

[kgodey]

Could have solved database switching better if navigation had been handled globally rather than focusing on the schema view.

Is there anything we could have done better to anticipate that issue? Are there any process improvements you'd like to make?

I think it's working fine, can't think of any improvements at the moment. We might try asking clarifying questions to other posts in order to dig further into what they are doing.

Everyone is encouraged to respond to other people's posts. I usually try to do it.

No concerns, I think it's a good decision. I'm wondering if we need to revisit the first milestone for read-only tables.

See above re: read-only tables

What do you mean by "revisit"?

[ghislaineguerin]

Is there anything we could have done better to anticipate that issue? Are there any process improvements you'd like to make?

I think this mostly became an issue when we decided to show only one schema at the time, so it was a consequence of introducing more layers of navigation. I also think I was delaying solving this by having the placeholder bar, which I didn't explore further. I think our current process is fine and we shouldn't change it now unless this happens again.

What do you mean by "revisit"?

I was thinking we'd change the add row functionality but we discussed this during our call and I think we are fine with inserting the row in the visible area rather than bottom of the table.

[eito-fis]

What went well?

Getting to remove the legacy csv functionality felt good :)
Otherwise, the mutli-db work went relatively smoothly and I'm feeling more comfortable with that area.

What could have gone better?

Had to go back and fix issue with older PRs. Will try to clarify expected behavior earlier in the process so we don't have to do this again.

Is this weekly update useful for you?

Yes, having a retrospective is always nice to get an understanding of what happened in the last week, and the active work section always helps to inform my work.

Do you read everyone else's responses to the discussion or just the original post?

I read everyone's.

What portions of the weekly update do you find most/least useful?

Active work is really nice just to track what I'll be working on.

Can you think of any improvements to the process of collecting data for the weekly update?
Can you think of any improvements to other aspects of the weekly update process?

Nothing in particular.

[kgodey]

I always enjoy deleting code, too. :)

[pavish]

How did this week go?
What went well?

I got the virtual list component working smoothly. It had been bugging me from the start.
Discussing with Ghislaine and giving her a demo of the current UI, made things easier for both of us.

What could have gone better?

I did not focus entirely on the milestone, I was wavering between it and virtual list, styling etc.,. I think it was for the best, but it still affects our planning if I don't stick to it or update the milestone accordingly. I could have handled this better.

Feedback on weekly update process:
Is this weekly update useful for you?

Very useful.

Do you read everyone else's responses to the discussion or just the original post?

I read everything.

What portions of the weekly update do you find most/least useful?

I find it all necessary. I sometimes find the Retrospective is a bit redundant, but it documents what has actually been completed, so it's pretty useful.

Can you think of any improvements to the process of collecting data for the weekly update?
Can you think of any improvements to other aspects of the weekly update process?

None, at the moment.

2. Basic Filtering, Sorting, Grouping is due this week. @pavish, you're the main person with assigned issues. Any blockers?

Yes. I have been focused deeply on the spreadsheet layout. I would like to get it completed first.
I would also like to complete the infra work to get the client build setup.
This way @ghislaineguerin could actually view and use the UI and we would be able to better any UX concerns.

Sorting is already done as part of the same PR. Filtering and Grouping will have a bit of work, so I think the milestone might get delayed further.

Please re-review the Roadmap. Does the order of the current milestones and their associated deadlines make sense?

Yes. The order does. As mentioned above, milestone 2 will get delayed. @kgodey Maybe we could update it's deadline by 2 days, to 21st.

[kgodey]

Thanks! I'll update the deadline to the 21st. We have two more milestones due on the 26th, though – do we need to delay those as well?

[pavish]

We can get Milestone 3 done by 26th. Milestone 4 may get delayed.

[kgodey]

I moved Milestone 4 to the 28th for now.

[ghislaineguerin]

@pavish I also got a lot of value from our call, we should try to do it more often.

[mathemancer]

• How did this week go?
  Not my most productive, for personal reasons. However, I do feel like we made some progress on the checking flow, and it feels great to see some of the design and other pieces continuing to come together.
  • What went well?
    I was pleasantly surprised with how extensible the column and alteration code was. I think the current mode of using our custom MathesarColumn as the column that we read in the table model passed to the API will make it easier to customize things in the future.
  • What could have gone better?
    I kept having to jump into and out of work, which made building momentum tough.
• Feedback on weekly update process:
  • Is this weekly update useful for you?
    Yes.
  • Do you read everyone else's responses to the discussion or just the original post?
    I try to read both. That being said, while I always read whatever's there when I'm writing my response, I suspect I've missed some replies that occur later on some weeks.
  • What portions of the weekly update do you find most/least useful?
    I usually find the work plan the most useful. I sometimes don't find much of relevance to my work in responses (I think this is okay, and to be expected).
  • Can you think of any improvements to the process of collecting data for the weekly update?
    I wish there was a way to automate linking completed PRs / closed issues into the doc.
  • Can you think of any improvements to other aspects of the weekly update process?
    I think it would make sense to break the top-level post in the GH discussion into multiple posts. That way, we can reply to each "bullet point" (or group of bullet points) without needing to copy them to our post for clarity. This might also promote more back-and-forth discussion on some things, since there wouldn't be so many large text walls.
• Please re-review the Roadmap. Does the order of the current milestones and their associated deadlines make sense?
  I'm a little nervous about the deadline of August 9 for 07. Initial Data Types. While many of those issues are quite quick, some will require custom types for which we may run into roadblocks or require discussion to understand the details.
• Feedback on the infinite scrolling:

  • Does the plan above make sense? Do you have any concerns?
    I think the plan makes sense, with the caveat that I'm not sure whether any extra back end work is needed to support the front end changes. If that's the case, it may be that the deadline is a bit tight.

  • Are we missing any work that needs to be specced out or designed?
    To my understanding, we should probably try to at least have a canonical ordering for each table whenever possible. Given that the entire row should be usable as a sorting key (or at least it won't be possible to tell the difference if two identical rows "switch order"), couldn't we create some such canonical ordering in the back end and return it by default? The only problem will be performance for large tables. But, for those tables, we'd expect an index anyway.

    If we want such a canonical ordering, we should spec it and make it so.

[kgodey]

I wish there was a way to automate linking completed PRs / closed issues into the doc.

I also wish there was a way to automate the retrospective, it would make my life easier.

I think it would make sense to break the top-level post in the GH discussion into multiple posts. That way, we can reply to each "bullet point" (or group of bullet points) without needing to copy them to our post for clarity. This might also promote more back-and-forth discussion on some things, since there wouldn't be so many large text walls.

I was already thinking that it would make sense to split up the discussion into multiple top-level posts to make conversation easier. I'll do that next time. Also, GitHub has a "Quote Reply" button in case that makes things easier for you. On a related note, it does make it easier to read when the thing you're responding to is quoted.

I'm a little nervous about the deadline of August 9 for 07. Initial Data Types.

We'll revisit the deadline once we start work on the milestone. I'm worried about it too but I'd rather make a decision once we have more data on how it's progressing.

I think the plan makes sense, with the caveat that I'm not sure whether any extra back end work is needed to support the front end changes.

No backend work is needed at this time.

To my understanding, we should probably try to at least have a canonical ordering for each table whenever possible. Given that the entire row should be usable as a sorting key (or at least it won't be possible to tell the difference if two identical rows "switch order"), couldn't we create some such canonical ordering in the back end and return it by default? The only problem will be performance for large tables. But, for those tables, we'd expect an index anyway.

If we want such a canonical ordering, we should spec it and make it so.

This is on my plan to spec out (I have it covered under "We may need to sort rows by some field by default to make the experience of scrolling and updating rows at the same time smoother"). I don't think it's going to block the implementation, it's just something we need to handle before we ship the alpha version. I'll put it in a later milestone.

[kgodey]

How did this week go? What went well?

It went fine. We got a lot done, and it's nice to see things moving along. I think the design is starting to come together and we're making a lot of progress on development. I am glad we're going to be making infinite scroll work.

What could have gone better?

We've needed to make some major changes to milestone orders in the last couple of weeks, which is fine, but I'd like it to stabilize so that we can make sure to complete all required backend and design work to unblock the frontend for future milestones.

I would also like to find the time to work on code more. I didn't do that this week and I'm continuing to think about how to structure my time better.

Is this weekly update useful for you?

I usually know everything that's going on with everyone, so not very useful. I'm not the intended audience, though, and it seems like everyone else does find it useful. It is also nice to see how much work we're getting done each work.

Do you read everyone else's responses to the discussion or just the original post?

I read everything.

What portions of the weekly update do you find most/least useful?

The questions are the most useful for me, and I think updates are good to make sure everyone's informed about anything major that's happening.

I think the "Active Work" section is useful to make sure we're focusing on the right things and that everyone has enough to work on.

I don't find the "Retrospective" section that useful.

Can you think of any improvements to the process of collecting data for the weekly update?

I'd like to automate it, and it seems doable in a few hours with the GitHub API, but I'm not sure it's worth doing with so much else going on.

Can you think of any improvements to other aspects of the weekly update process?

I think splitting each section into its own post on the discussion will make things more readable and promote discussion. I'm also thinking about better ways to structure active work / retrospective to make it easier to read, but I don't have any concrete ideas yet.

[Dhruvi16]

• How did this week go?
  • What went well?
    I was able to push the second iteration of the button to review.
  • What could have gone better?
    I had an unproductive week due to vaccine side effects so would try and cover the pending work this week.
• Feedback on weekly update process:
  • Is this weekly update useful for you?
    Yes
  • Do you read everyone else's responses to the discussion or just the original post?
    I do try to read everyone else's response in order to not fall behind.
  • What portions of the weekly update do you find most/least useful?
    I find the retrospective section redundant.
  • Can you think of any improvements to the process of collecting data for the weekly update?
    As @kgodey mentioned, we can automate it as we do all our work on Github.
  • Can you think of any improvements to other aspects of the weekly update process?
    None
• 02. Basic Filtering, Sorting, Grouping is due this week. @pavish, you're the main person with assigned issues. Any blockers?
  None
• Please re-review the Roadmap. Does the order of the current milestones and their associated deadlines make sense?
  Yes it does
• Feedback on the infinite scrolling:
  • Does the plan above make sense? Do you have any concerns?
    Yes, the plan makes sense. No concerns.
  • Are we missing any work that needs to be specced out or designed?
    None at the moment.