2022-01-31 weekly planning

[kgodey]

About

At the start of each week, we start a discussion covering Mathesar's progress, team member check ins, and general updates. Everyone is welcome to add a new topic to discuss or to comment on existing topics.

Discussion topics

• Last Week in Mathesar
• Frontend priorities
• Backend priorities
• Design priorities
• Product priorities
• Comms Assignee schedule
• Google Summer of Code project ideas
• Dominykas' check-in
• Brent's check in
• Sean's check in
• Ghislaine's check in
• Kriti's check in
• Pavish's check in
• Mukesh's check in

Weekly check-in

@centerofci/mathesar-core please post your weekly check-in as a top-level response and edit the discussion topics above to add your check in as a topic. Others are also welcome to respond.

Prompts:

• How did your most recent work go?
• What are you planning to work on soon?
• Do you anticipate any dependencies or blockers that someone else on the team can help you with?
• Our progress towards shipping new features seems to have slowed significantly in the last few weeks. Are you also noticing this and if so, do you have any ideas about how we can ship features faster?

If you'd like to add anything else to your check-in that's not related to the prompts, feel free to do so.

[kgodey]

Last Week in Mathesar

Feedback Requested

• Discussions in progress:
  • Views Product Spec #1015
  • Code Styling guidelines for the backend #1007
• Specs in review:
  • None at the moment
• PRs needing review:
  • Please see frontend and backend priorities sections.

Complete

• Product specs finalized:
  • Product spec for Views mathesar-wiki#22
• Milestones closed:
  • 2022-01 improvements with 15 issues complete.
• Major wiki updates:
  • The Community page has been revamped. We now have sections for different community forums:
    • Matrix
    • GitHub
    • Mailing Lists
    • Events
      • 2022-01-22 Code for Japan Social Hack Day notes
  • We now have better documentation for filters.
    • "Filters" Concepts page
    • "Filters in Mathesar" engineering architecture page
• PRs merged:
  • Backend: None
  • Frontend
    • Add test for filterUtils and CancellablePromise #1013
    • Check for TypeScript errors in our CI workflow #1025
    • Improve line-height CSS #999
    • Use more strict export for proceed function within CancelOrProceedButtonPair component #1009

[kgodey]

Frontend priorities

1. Get Type configuration for Text, Number, Boolean, and all mathesar types with single associated db type #926 ready for review and merged.
2. Review and merge PRs ready for review.
   1. Display null while editing a null cell #961
   2. Keyboard Accessibility Pagination Component #963
   3. Implement renaming a column #976
   4. UI for adding multi-column unique constraints #1005
   5. In-app help content #1024
3. Continue to make progress on the Working with Tables milestone.
   1. Merge pending PRs for the following issues:
      1. Implement renaming a column #497
      2. Frontend work for adding multi-column unique constraints #531
   2. Implement setting a default value for a column #500
   3. Improve UX when adding new records to a table with a NOT NULL column #775 should hopefully be unblocked by early next week
4. Write integration tests for some important user scenarios #640
5. Other issues from the Initial Data Types and 2022-02 improvements milestones. Some ideas:
   1. Create Menu component #742
   2. NumberInput component improvements #970
   3. Wait until user is done editing a cell before saving #928
   4. Blur via keyboard after entering edit mode on a NULL cell should cancel edits #929
   5. Add keyboard shortcuts for setting a cell to NULL #936
   6. Add ability to minimize and expand groups in the frontend #475

@pavish @seancolsen Please comment below with thoughts/concerns.

[seancolsen]

• Point 4. above is my main priority right now.
• Display null while editing a null cell #961 (within point 2.) is also on my list.
• There's nothing else I can work on within 1. - 4..
• Within 5., I think Create Menu component #742 is highest priority, so I'd start there.
• I'd like some feedback from @kgodey and @pavish on how much to prioritize 4. vs 5..
  • How much integration test coverage do we want?
  • After I improve the process of writing integration tests (as I'm working towards with Improve E2E test setup #1032), I could easily spend a whole week writing integration tests. If we really want an E2E test for each user scenario (as Write integration tests for some important user scenarios #640 describes), that might take more than a week, depending on how granular we get. And some of those tests will break as we continue changing the UI. Testing is an artful and strategic balancing act. It seems really important for us to have some E2E test coverage. But at our current early stage (and with our desire to increase velocity), I'd lean towards aiming for sparse E2E tests right now. I'm not sure how to define that specifically though. Perhaps we define it by saying "spend 2 days writing E2E tests"? I'm open to suggestions here.

[kgodey]

@seancolsen As I understand it from previous discussions with @pavish, the overall goal of integration testing is to help us move faster while developing frontend features with the confidence that we haven't broken anything (i.e. increase frontend velocity). For your current work, I think it makes sense to have enough integration testing to help future velocity and no more. I don't know enough about the details to figure out how many tests that would be but I assume you and @pavish can work it out?

After you're done, it would be great to create tickets with a list of additional tests that could be written, it could be a good way to get new contributors into the codebase or it could be a good Google Summer of Code project for someone.

[pavish]

@kgodey Looks good to me.

Regarding point 5, I'd say the priority would be #970. The rest can potentially wait (and some are good first issues) until after we complete implementing working with all data types on the frontend.

I think we are all in agreement over integ tests:

• Keep E2E tests sparse and focused only on functionality rather than appearance. We do not have to write granular tests atleast until the alpha since our UI is bound to change frequently. @seancolsen and I will come up with the essential test list.
• Write integration tests for some important user scenarios #640 is a meta issue. We'll have to multiple issues for other user scenarios and specify them in this issue.

[silentninja]

I would like to suggest using BDD and gherkin, it can give a birds-eye view for everyone to take a look at instead of having to delve into the details. Moreover, it makes us think in terms of features instead of logic. I can help with e2e testing as I have good experience with it.

[pavish]

I'm moving #500 to the data types milestone since the groundwork for it basically includes implementation of data type switching and type specific user input.

[kgodey]

Backend priorities

1. API error handling: I've cleaned up the meta issue and I think we should use it to track further error related issues that come up. Only these three PRs should be prioritized at the moment:
   1. Convert API layer error responses into Errors that conform to the API spec #991
   2. Fixes #859 Return proper error response on not null violation #1017
   3. Handle error consistently on the backend using custom drf error handler #1016
2. Filtering:
   1. Get Introduce function API #1022 merged
   2. Create PR for new filters API.
   3. Close outdated PRs:
      1. Backend text filtering #879
      2. Backend email filtering #893
3. Set up style guide and tooling for consistency in backend code #1008
4. Continue to make progress on other Initial Data Types issues. Some ideas, in no particular order:
   1. Data Types:
      1. Handle INTERVAL type in the backend #430
      2. Handle MONEY data type in the backend. #420
      3. Backend work for Mathesar money custom type  #417
   2. Spec for human readable date and time parsing #845
   3. Replace column name usage with ID:
      1. Replace column index usage with attnums and Column model id #839
      2. Modify records API to use column id instead of column name #896
      3. Modify constraints API to use column id instead of column name #898
   4. Grouping:
      1. Implement grouping for Text types. #408
      2. Implement grouping options for the email type #411
      3. Implement grouping options for the URI type #414
      4. Implement range based grouping for numeric types. #384
      5. Implement grouping options for date & time types #429

@dmos62 @mathemancer @silentninja Please comment below with thoughts/concerns.

[silentninja]

@kgodey I moved issue #1017 a step higher as it is blocking the frontend and easier to review(not sure if you listed based on priority). Should #839 be worked on immediately as it comprised of good first issues?

[kgodey]

Design priorities

1. Design specs for:
   1. Support for opening and interacting with a View #466
   2. Design for List data type #978
   3. Improve support for finding duplicate rows #1028
2. Continue exploratory work for Views in the design prototype.

@ghislaineguerin @ppii775 Please comment below with thoughts/concerns

[kgodey]

Product priorities

1. Update remaining Views design issues with requirements based on the new product spec for Views
2. Finalize API support for storing "record preview" and "record selector" settings #944

[seancolsen]

I'll take responsibility for facilitating the finalization of #944.

[kgodey]

Thanks @seancolsen!

[kgodey]

Comms Assignee schedule

Week	Person
2022-01-31 to 2022-02-04	Mukesh
2022-02-07 to 2022-02-11	Sean
2022-02-14 to 2022-02-18	Dominykas
2022-02-21 to 2022-02-25	Ghislaine
2022-02-28 to 2022-03-04	Brent
2022-03-07 to 2022-03-11	Kriti
2022-03-14 to 2022-03-18	Pavish

Instructions

You can also find the schedule and instructions for the Comms Assignee on the wiki page. Please feel free to update the wiki page if you think of anything useful during your shift.

[kgodey]

Google Summer of Code project ideas

The Google Summer of Code 2022 timeline has been announced. Organization applications open Feb 7. Since we plan to participate, this means we need to get our project ideas in shape pretty soon.

Please put your project ideas on the wiki page ASAP. Post here or DM me if you have any questions.

[dmos62]

I'm having trouble coming up with GSOC project ideas. Would anyone care to share theirs? They can be rough, I want to get on the same page.

[kgodey]

@dmos62 We came up with some ideas in the State of Mathesar meeting, I think you had left by then. See here.

[dmos62]

@kgodey I added an idea to the wiki page. I didn't fill out the details, since I'd like to not jump into detailed planning before there's some agreement that it's a good idea.

[kgodey]

@dmos62 I think that's a good idea. Please go ahead and add the details!

[dmos62]

@kgodey I added details. Notice that my difficulty and time estimates are somewhat conservative.

[dmos62]

How did your most recent work go?

I'm pretty happy with the changes in how filtering work is being organized (smaller PRs). That was largely enabled by finding out how filtering will look, in rough terms.

What are you planning to work on soon?

Make the requested changes on the functions API PR; create the abstractions API namespace; create a filters endpoint in the abstractions namespace; see if it's the right time to make some improvements to the abstractions API ( named parameters and generics); document; start adding filters required by spec.

Do you anticipate any dependencies or blockers that someone else on the team can help you with?

No.

Our progress towards shipping new features seems to have slowed significantly in the last few weeks. Are you also noticing this and if so, do you have any ideas about how we can ship features faster?

I am noticing this. I hope that the pace will pick up from my end in the near term. Especially since my planned tasks are less exploratory and more defined. I'm also trying out some changes in my routine.

[mathemancer]

Brent's check in

How did your most recent work go?

Quite slowly; more on that below.

What are you planning to work on soon?

I have a short week due to the Lunar New Year, but I'm planning to continue work on the Interval type, PR reviews, and hopefully some grouping.

Do you anticipate any dependencies or blockers that someone else on the team can help you with?

None that will be an issue this week. Rather, I expect I'll be the one holding things up as I'll be in and out with family events.

Our progress towards shipping new features seems to have slowed significantly in the last few weeks. Are you also noticing this and if so, do you have any ideas about how we can ship features faster?

I agree. I think we've adopted an "architecture first, features second" culture in the back end. This has some advantages, but a disadvantage is reduced velocity at producing features.

• I'm guilty of this, so not pointing fingers at anyone that I'm not pointing at myself.
• We should try to remember that it's not clear yet what the final form of the product or API will be, so trying to design things so that we can achieve some ultimate end may be wasted effort.
• As an example, I spent a large part of last week trying to do the interval type the "right way"
  • psycopg2 hooks up a python type to the underlying DB INTERVAL type that loses information.
  • The "right" solution would mean writing a custom translator and loading it globally using psycopg2, then modifying the Interval class in SQLAlchemy to use the right custom python class to represent the DB interval type, etc.
  • The solution I went with in the end is a bit of a "hack". I just bypass all of that and set up SQLAlchemy to preprocess intervals by casting to a string in the appropriate format on the DB, then treat the column as a string in python.
  • The hack took 15 minutes to implement.
  • Researching the "right" solution took multiple days, and the implementation would have taken another few days and required (potentially, I stopped looking into it) compiling custom versions of some psycopg2 C libraries.
• The focus on architecture also has the tendency to lead to lots of conversation, since decisions are expected and assumed to be wide-ranging and long-lasting in their effects, making team-wide buy-in important.
  • If we can focus on making sure that changes are smaller in their effect, and assumed to be "temporary", then we can probably avoid some conversation.
  • The downside would be that we will have to redo things once some temporary solution or hack fails to scale or extend easily.

There are some major downsides to avoiding thinking too much about the big picture architectural things (see the problems we're having with connections and engine handling for an example). But I do think that focusing on that stuff is slowing us down a lot on the back end.

[kgodey]

Your analysis makes a lot of sense. I think adopting a "features first" approach makes sense, even if we do have to redo things down the line, because there might not be a "down the line" at all if we don't prioritize features and getting Mathesar into users' hands. :). I don't think this only applies to the backend given @seancolsen's notes. I'm not sure how to help enforce/encourage this approach, any ideas?

cc @dmos62 @pavish @seancolsen @silentninja.

[silentninja]

I think adopting a "features first" approach makes sense, even if we do have to redo things down the line, because there might not be a "down the line"

@kgodey You mentioned that we shouldn't be reopening issues when I mentioned we can ship imperfect features if necessary and reopen create a new issue if a better solution comes down the line. But this comment seems to be okay with it, has the opinion changed?

[silentninja]

@kgodey I made a few suggestions in my check-in, would love to hear comments on it.

[kgodey]

@silentninja I think I misunderstood you initially, what I meant in that conversation was that we shouldn't keep rethinking implementation before we're shipped a feature. I'm absolutely fine with shipping imperfect features and then improving them (I would create a new issue instead of reopening an old one, though).

Thanks for the suggestions! I've been reviewing everyone's ideas and I'll figure out action items soon.

[seancolsen]

Sean's check in

How did your most recent work go?

• I feel good about my progress last week.
• I expected the E2E testing stuff to take a while. It's tricky work because it touches a wide swath of our architecture. I'm aiming to get to a point where we have enough tooling and documentation in-place that front end devs can easily write integration tests. My draft PR makes very little actual coding changes, but it took a while to figure out what needed to be changed. There's still some more to do, but I'm excited about the direction it's going. I'll feel much more confident digging into refactoring projects once we have some E2E test coverage.

What are you planning to work on soon?

• See Frontend priorities

Do you anticipate any dependencies or blockers that someone else on the team can help you with?

• Nope

Our progress towards shipping new features seems to have slowed significantly in the last few weeks. Are you also noticing this and if so, do you have any ideas about how we can ship features faster?

• Yeah I have noticed this too.
• See "Ideas for moving faster" below

Personal updates

• My kids are still home from school. This covid wave has been grueling. Cases are still rising here in Knoxville. Cold weather makes it hard to play outside too. The kids are getting really antsy and irritable, which has meant a lot of distractions for me. I really really want them to go back to school!

Sean's ideas for moving faster

⚠ Disclaimer: I'm not attached to any of these ideas. I suggest them only as brainstorming in response to the prompt, and I don't want to get bogged down in a time-consuming debate about how to spend less time in time-consuming debates.

Front-end-specific

• @pavish has explained that the long-term goal of our front-end architecture is to build and release: (1) a general-purpose CSS library, (2) a general-purpose component library which makes use of the CSS library, and (3) the Mathesar UI which makes use of both the CSS library and the component library. That's a lot of general-purpose code, and a lot of concerns to take into account besides those of Mathesar. I would be able to move faster on Mathesar features without those additional goals. However, it's also worth mentioning that at this point it would take some extra work to simplify our architecture to the point where I'd see all the gains from that simplification.

Design specific

Our design process could be streamlined and still give me all the design spec that I need to be productive at implementing the design.

Note: I suggest these ideas cautiously, with the caveat that I'm speaking only about what I need to do my work. Others may need more from our design process than I do. I speak only in the spirit of brainstorming -- not as a request.

What follows is a description of how I'd approach our design process if it were up to me.

• At the product design phase, I'd write a list of goals from the user's perspective. I think it could be quite a bit more terse than we currently produce. Kriti would write the product designs and a "Product team" would review them. The Product Team might include Kriti, Ghislaine, Pavish, Brent. Others could join the Product Team if they want. But those not on the team would not spend any time on it or receive any notifications about it.
• At the UX design phase, I'd do it primarily in text, by writing a few user stories for each product goal. I wouldn't do a prototype. For some interfaces that are more cumbersome to describe in text, I'd supply a very rough wire-frame diagram, being clear that it does not convey visual design. A UX Team would review it, including Kriti, Ghislaine, Pavish, Sean.
• I'd make it clear (perhaps it already is?) that visual design should be done via improvisation by the front end dev during implementation.
  • After we perform our UI styling, then I'd alter our process to supply a visual design before dev work begins.

Team-wide

I think I got the gist of my ideas across at the last State of Mathesar meeting. I'd summarize by saying:

• Provide structures to facilitate more synchronous discussions.
• Reduce the number of people involved in decisions.
• Reduce unnecessary notifications.

A few more thoughts on reducing notifications.

• I find the "Code - Review" Matrix channel strange. It seems redundant with the notifications that people should be getting in GitHub. I have a vague memory of Kriti explaining that we use the channel to allow outside contributors to see PRs that need review. If that's the case, then I don't think that's a good-enough reason to have the channel. I also get the feeling that the channel may exist in-part to bring more attention to PRs that need review, even when people who should be reviewing them have already gotten notified. If that's the case, then I think it would be more effective to draw attention to those notifications by reducing other notifications.

[kgodey]

I'd write a list of goals from the user's perspective. I think it could be quite a bit more terse than we currently produce.

With regards to the product specs, the list of goals is the easy part. Most of my energy is actually spent on figuring out how to translate user goals to PostgreSQL actions and concepts; this is also the reason for the length of the specs. I think for many products out there, we can define user goals and figure out implementation details of those goals down the line, but because we have the goal of everything we do also be reflected outside of our software in another piece of software (PostgreSQL), the implementation details and the user goals are not easily separable. I'm not sure how to streamline that process, but if you had any ideas or examples that would make it easier, please let me know.

[pavish]

That's a lot of general-purpose code, and a lot of concerns to take into account besides those of Mathesar.

I think we are at a point where we do not have to regularly implement new general-purpose components. The essential ones are already in place.

I would be able to move faster on Mathesar features without those additional goals.

Moving forward, how about we implement them as mathesar-specific components first and then move it to the general component-library later. I hope this should make this easier for you by not having to focus on the long-term goals.

[ghislaineguerin]

Ghislaine's Check-in

How did your most recent work go?

It went well, although the larger scope didn't help with focus. I'm glad to have had a conversation w Kriti and narrowed down the options.

What are you planning to work on soon?

I plan to continue work on #466 and have the spec reviewed. Also, work on #1028.

Do you anticipate any dependencies or blockers that someone else on the team can help you with?

Outside of changes to Views Product Spec, none.

Our progress towards shipping new features seems to have slowed significantly in the last few weeks. Are you also noticing this and if so, do you have any ideas about how we can ship features faster?

I do notice it. From a design perspective, it helps to have more discussions w product (Kriti)

[kgodey]

@ghislaineguerin FYI, I made a small update to #1028 (see the "Design notes" section) that I forgot to include earlier.

[kgodey]

How did your most recent work go?

Figuring out Views has involved taking two steps forward and one step back multiple times, including last week. I started off with the assumption that (PostgreSQL) views were the solution to a bunch of user flows and have been slowly realizing that they are not, which involves walking back a bunch of decisions. I don't think this is a failure, it's how iteration is supposed to work (and it's only been affecting design work), but it still feels frustrating personally.

What are you planning to work on soon?

• A bunch of updates to the Views spec and design tickets, including fleshing out a new query builder interface that can be accessed from both tables and views.
• Figuring out how to increase feature velocity
• Backend style guide and tooling

Do you anticipate any dependencies or blockers that someone else on the team can help you with?

• Thoughts about how to speed up feature velocity would be useful, see below.

Do you have any ideas about how we can ship features faster?

As @mathemancer and @seancolsen pointed out, we're spending too much time on perfecting architecture. We should figure out how to re-focus on getting features out as specced and improving architecture later. I'm open to ideas on how best we can achieve this, it would be helpful if everyone could think about what they need to improve velocity in their own work and I can figure out how to make that happen.

[pavish]

Pavish's check in

How did your most recent work go?

I was away for a couple days last week and mostly focused on reviewing PRs and discussions. The PR for type configuration is ready for review as of today.

What are you planning to work on soon?

• Finalize discussion on Dynamic Type selection & customization on the frontend #1035 and get Type configuration for Text, Number, Boolean, and all mathesar types with single associated db type #926 reviewed and merged
• Review and merge open frontend PRs
• Work on NumberInput component improvements #970
• Create issue and work on CellInput component

Do you anticipate any dependencies or blockers that someone else on the team can help you with?

None.

Our progress towards shipping new features seems to have slowed significantly in the last few weeks. Are you also noticing this and if so, do you have any ideas about how we can ship features faster?

Yes, I agree.

• I concur with both @mathemancer and @seancolsen that we are spending more time on perfecting architecture. I'm willing to cut some small concerns to get Mathesar out first.
• We seem to be working more on improvements of existing features in the last few weeks than implementing new features. I think if an improvement can be implemented later, we should delay it for later and only work on essential improvements.

I major problem I'm facing is that, in the last month, my development time has reduced significantly. I've been focused primarily on PR reviews, discussions and planning out architecture. This has lead me to:

• Having to switch context frequently. I am however, trying to have dedicated focused days as we talked earlier on our team's sync call.
• I usually have a set of PRs opened to review regularly and a number of them involves me having to think about implementation architecture. This takes considerable time. This cannot be avoided though.
• I am also guilty of "architecture first, feature second" approach. But in certain cases like type configurations, I am not really sure if we can avoid that.

[silentninja]

How did your most recent work go?

It went well, though it took some time, really happy that Error handling related PR's are close to completion.

What are you planning to work on soon?

• Focus on migrating API from using Column name to id on discussion days or when thinking about the product. I will be starting with Modify records API to use column id instead of column name #896
• Grouping issues starting with Implement grouping for Text types. #408

Do you anticipate any dependencies or blockers that someone else on the team can help you with?

None

Do you have any ideas about how we can ship features faster?

1. Separation of concerns - Though we have split the codebase. our PR's are still coupled. We are not treating our modules as separate libraries with api's to join them, resulting in PR's being huge or not focused which further results in less parallel work. Having small issues will help us in getting more contributors introduced to our codebase.

2. Priority - Our priorities have been moving back and forth between perfecting an architecture and releasing a feature based on the architecture. Since we are at an early stage this is inevitable but we can solve this by making certain compromises like doing certain things manually, leaving out certain cases that would put us on the bad side of the 80-20 rule. Though this will introduce technical debt and go against our concept of a linear milestone, it is worth considering.

3. Linear Milestone - Currently our milestones are linear, so this makes our work aim for perfection. We should rather aim to have an extended(or future) counterpart for the milestone. While we have it various forms as of now(future tags, future ideas wiki page), it would be much better if we have it as a milestone, as it would make us actually use it and think about it when working on day to day basis,

4. Detailed proposal - Our issues are quite abstract(as it seems to come from a features standpoint) and do not delve into the details, which is fine(Tbh I find milestone issues to be quite concise) but this leads to PR's trying to address various concerns within a single PR, we should be breaking up into Architecture required to support the feature, necessary details required for the feature, interface/api for the feature. I find the current decision to split the apis into two a good step in the right direction, as it could help us prioritize, estimate better, have fewer blocking issues. Right now this is seen as a best practice, but would be good to enforce it.

5. Behaviour Driven Developement - Having BDD style test for UI apis and unittest for db and logic testing of UI apis would help us focus better on features rather than architecture, creating a good separation of concern.