Tasty Bugs

Checkbox Black Hole

Issue

• I have a checkbox within a disabled fieldset. Like other elements in the fieldset, the checkbox input inherits the disabled attribute from the disabled parent.
• My goal was to have the not-allowed cursor appear any time the user clicked a disabled element. The following code was sufficient for all inputs except the checkbox:

  *:disabled:active { 
    cursor: not-allowed;
  }

• After extensive trial and error and some research, I discovered that several click events and the :active CSS psuedo-class did not register for a disabled checkbox. I finally came to the conclusion that I would need an alternative to the onClick, onMouseDown, and onMouseUp events.
• I tried wrapping the checkbox in a div but only the edges of the element would be triggered with any of the aforementioned techniques. Hence why I regarded the checkbox as a black hole since it seemed to sit on top of its wrapper and reject any click events.
• I knew that it still register client actions, however, since :hover and onMouseOver worked so I kept searching to find some means of emulating the :active pseudo-class.

Resolution

• I discovered the onPointerDown and onPointerUp events which had functionality mirroring onMouseDown and onMouseUp. The following code enabled me to capture and utilize the client's click in its entirety:

  .not-allowed, *:disabled:active { 
    cursor: not-allowed;
  }

  const [clicked, setClicked] = useState(false);
  ...
  <input
    ...
    onPointerDown={() => setClicked(setting.id)}
    onPointerUp={() => setClicked(false)}
    style={{ cursor: setting.id === clicked ? 'not-allowed' : '' }}
  />

---

Scrapy Spider Settings

Issue

• print(get_project_settings().attributes) in app/api/routes/search_routes.py prints a settings object with default keys and values.
  • No matter how I alter the settings.py file (with a path of app/crawler/spider_lair/settings.py), the changes are not reflected in the aforementioned print statement.
  • The Scrapy spider also reflects no changes in the settings.py file.
• Initially, I was only concerned with a particular setting, the spider's timeout. I came up with an interesting attempt at a workaround:

  """Spider timeout.
  
  Start time is taken from spider's stats. Though it's congruent
  with UTC, it's naive and thus must be localized. Current time is
  set to UTC ab ovo. With two aware datetime objects and identical
  timezones, the difference in seconds can accurately be found.
  """
  
  from datetime import datetime, timezone
  import pytz
  
  # class' try block begins
  start_time = pytz.timezone('UTC').localize(self.crawler.stats.get_value('start_time'))
  current_time = datetime.now(timezone.utc)
  diff_in_sec = (current_time - start_time).total_seconds()
  
  if diff_in_sec >= 10: raise CloseSpider(reason='timeout')
  # class' try block ends

  • One large issue with this, however, is that the if block raises an exception which triggers my except block and loses the spider's yield. So, albeit the time logic works, the exception, lack of DRYness, and the later discovery of the more deep seated issue, all rendered this workaround unacceptable.

First Fix

• A workaround that was successful in targeting particular settings but fell short of cleanly addressing the underlying problem was the following:

  custom_settings = {
      'CLOSESPIDER_TIMEOUT': 10
  }

  • I declared this as an attribute of the class, and it successfully stopped my spider after ten seconds with no additional code needed to bring this about. For this to completely address the issue, however, would mean placing every altered setting into this dictionary and placing that attribute in every spider that was created.

Resolution

• Research informed me that CrawlRunner had to be explicitly passed Scrapy settings. I tried several ways of passing the settings.py file in app/crawler/spider_lair but was unsuccessful in getting the crawler to register anything else other than the default settings.
  • Since I was running my crawler programmatically from another directory, I thought perhaps it could be related to where my module paths were pointing. I altered them at app/crawler/spider_lair/settings.py and app/crawler/scrapy.cfg, to no avail.
  • I finally decided to make a settings.json file in app/api/routes. Inside, I placed all the default settings that I changed and then updated the Scrapy settings in app/api/routes/search_routes.py and passed it to CrawlerRunner:

    import json
    
    settings = get_project_settings()
    settings_dict = json.load(open('app/api/routes/settings.json'))
    settings.update(settings_dict)
    crawl_runner = CrawlerRunner(settings)

  • Initially I hit ModuleNotFoundError: No module named 'spider_lair' which I rectified by removing the following from the JSON file:

    "SPIDER_MODULES": ["spider_lair.spiders"],
    "NEWSPIDER_MODULE": "spider_lair.spiders",

• This was completely successful, and it opens several opportunities for ephemeral setting manipulation by the client (if it can be read, it can be written to).

---

Finding the Favicon

Issue

• The only page that shows my favicon on the initial DOM load is my homepage. Every other page only shows it once a user has navigated to the homepage first. How can this be rectified?

Resolution

• Consider eliminating this line in public/index.html:

  <link rel='icon' type='image/png' href='./favicon.ico'>

  which throws off the data being served in app/__init__.py:

  @app.route('/', defaults={'path': ''})
  @app.route('/<path:path>')
  def react_root(path):
      if path == 'favicon.ico':
          return app.send_static_file('favicon.ico')
      return app.send_static_file('index.html')

---

Browser Caching

Issue

• When I’m on my site and then navigate to another page off my site and then click the back arrow to access my site again, only JSON is returned, I have to refresh just to see the page again. Any idea how to address this?

Rumination

• It must be something to do with the headers of my response. I’ve scoured my front and back end and researched extensively and there are a few behaviors that have quite telling:

  • When I navigate from my auth page to another page and then click the back arrow, it throws a Unexpected < at position 0 in JSON error.
  • When I navigate from my history page to another external page and back, it returns pure JSON with the header of the response being application/json.
    • When I refresh the page, however, all my content comes back and the following response header is returned for that page’s GET request: Content-Type: text/html; charset=UTF-8.

• What this all tells me is there’s dissonance between data format types. My view needs HTML; however, external access of my site through history traversal is resulting in a JSON object being returned.

• Another individual that I discovered went through a similar occurrence had this to say about it: Credit to Jonathan Creamer for Chrome back button in express caching issue solved:

   Welp, turns out, Chrome doesn't take the headers into
   account for caching unless you tell it to. It usually
   only will use the URL and the request method (get, put,
   etc). In order for Chrome to cache differently based 
   on the content/type, you have to add a header...
  

  app.use((req, res, next) => {
  	res.header("Vary", "X-Requested-With");
   	next();
  });

  This will allow chrome to cache both the HTML and
  the JSON pages separately and prevent the issue 
  from happening.
  

• This alongside other information gleaned from research truly makes me think there is a header/cache issue that must be resolved.

• I was thinking at one point to merely load the page if the user accessed my site from an external site via history by way of something like

  const performanceEntries = performance.getEntriesByType('navigation');

  • However, none of my front-end code is being hit when this happens. I can't get a console.log through, let alone an event handler utilizing data derived from that method.

• My back end was not reporting any request coming through, successful or otherwise. I watched for any status code updated and put a print statement to try to see if there was any information whatsoever I could leverage to my advantage.

• This did does not occur when navigating between pages on my site, or reloading my DOM. Oddly enough, it doesn't occur on my search page either.

• Here are my responses annotated (I'll delimit the comments by a //$ or //!): When Traversing History

  General
  -------
  Request URL: http://localhost:3000/api/history/ //$ request sent
  Request Method: GET
  Status Code: 200 OK (from disk cache) //! note the source
  Remote Address: ...
  Referrer Policy: strict-origin-when-cross-origin
  
  Response
  --------
  access-control-allow-origin: *
  Content-Encoding: gzip
  content-type: application/json //! note the content type
  date: Thu, 18 Nov 2021 08:52:11 GMT
  server: ...
  Vary: Cookie, Accept-Encoding
  X-Powered-By: ...
  

  When Reloaded

  General
  -------
  Request URL: http://localhost:3000/api/history/
  Request Method: GET
  Status Code: 200 OK
  Remote Address: ...
  Referrer Policy: strict-origin-when-cross-origin //$ same as last
  
  Response
  --------
  HTTP/1.1 200 OK
  X-Powered-By: ...
  Accept-Ranges: bytes
  Content-Type: text/html; charset=UTF-8 //! note the content type
  Content-Length: 376 //$ length now present
  ETag: W/"178-OsCzVrvIeBl2Rz3NCoe67UslmKY"
  Vary: Accept-Encoding //$ similar to last
  Date: Thu, 18 Nov 2021 08:52:10 GMT
  Connection: keep-alive
  Keep-Alive: timeout=5
  

• I thoroughly tested this on Safari and could not recreate the bug which is yet another indication that it's Chrome's caching protocol that's responsible. I feel I'm closing in on this bug's radix.

  • Side note, when testing on Safari, I noticed that all of my navigation bar elements were mirrored. It's interested to see how different browser default arrangements/styles are. Overall, however, everything appeared to be working smoothly on there.

• I discovered an excellent article called How to Solve Caching Conundrums by Craig Buckler, outlining my exact situation and confirming a lot of the ideas I had arduously garnered over my time researching. He states the fix is the following:

  The fix: ensure your page and data URLs are never the
  same. When navigating to http://myapp.com/list/?
  search=bob&page=42, the Ajax call should use a
  different URL: it can be as simple as http://myapp.com/
  list/?search=bob&page=42&ajax=1. This ensures Chrome
  can cache both the HTML and JSON requests separately,
  but JSON is never presented, because the Ajax URL
  never appears within the browser address bar.
  

  • This led to the recrudescence of the 404 on First Render I had quelled. It resulted in a 404 on the actual page when navigating backwards, a step in the wrong direction for my situation it seems.
  • Albeit that fix did was not effective in my case, the article was invaluable for both confirming ideas I had developed and informing me more about the topic at hand.

First Fix

• After extensive research and testing, I found that I had to modify the way the Chrome browser cached by constructing a custom response in my back end:

  """History Routes.
  
  The method make_response allows for the decoration
  of my response object. Here I use it to add the headers
  I need.
  """
  
  from flask.helpers import make_response
  
  response = make_response({ 'history': [ entry.to_dict() for entry in entries ] })
  response.headers['Cache-Control'] = 'no-cache, no-store, must-revalidate'
  return response

  • The wonderful aspect of this solution is that it doesn't force a reload when navigating between pages on the site, only external sources entering. So, for day-to-day actions, there will be no performance hit and as for users entering from an external origin, this could be seen as an extra layer of security for the site.
• How do we control web page caching, across all browsers?, How to prevent cached response (flask server, using chrome), Disable cache on a specific page using Flask, and make_response were the final resources that allowed me to connect the dots and rectify this part of the bug. This gives a nice description of the differences between some of the header values Why both no-cache and no-store should be used in HTTP response?.
• The following error was persisting on my login page:

  Uncaught (in promise) SyntaxError: Unexpected token <
  in JSON at position 0
  

  • I discovered the Chrome browser was refusing to drop the cache for that page and so I wrapped my async dispatched in a try/catch block to force a reload should any HTML make it into the response.

     useEffect(() => {
    (async () => {
        try {
            await dispatch(authenticateLogin());
            await dispatch(authenticateSignup());
            setLoaded(true);
        } catch (err) {
            window.location.reload();
        }
    })();
    }, [dispatch]);

    • Note that this statement does not make the other facet of the solution superfluous. If one removed the headers from either back-end route connected with these dispatches, only the JSON object {"errors": ["Unauthorized"]} is observable in the front end.
    • Fortunately, albeit wrapped in the try/catch statement for semantic reasons and more security, the await dispatch(authenticateSignup()); is not invariably affected by the reload which has only been triggered thus far by the aforementioned syntax error. I confirmed this by placing an alert() in the catch block.
• I later discovered that the bug persisted on the deployed site and that it had grown in scope, now affecting all page refreshes.

Resolution

• It turned out that this error was connected with the 404 Error on First Render. The front-end (App.js) routes and the back-end (Flask) routes should not be congruent. This leads the browser to serve the wrong type of data when reaching that URL. So, I left my back-end routes with /api and removed that from my front-end routes, here's an example:

  <!-- Redux Front-End Route -->
  '/api/search/results/'
  
  <!-- Flask Back-End Route -->
  '/api/search/results/'
  
  <!-- App.js Front-End Route -->
  '/search/results/'

• As for the 404 that was raised in the console upon first render of each page alongside the 404 that was raised on the page when navigating to an external page and back to my site, none of that is present on the deployed site. So, localhost was giving me a false positive.
• Since the first fix had patched the bug locally, every test had to be pushed up to GitHub Actions and then observed on Heroku which significantly increased the amount of time it took to address the bug.
  • I would scour its network tab and compare it to other projects, trying to find what could be going awry.
  • I reached out to others and that's when it was suggested that I return to my previous route arrangement. A resource shared to me which detailed a situation to mine (React - Page doesn't display when using the browser back button) also suggested that the issue may be arising from route congruence.
  • I changed the routes and upon testing it again, I discovered that this bug didn't manifest in either the console or the view on the deployed site.

---

Search Page Permissions

Issue

• Refreshing my page while not logged in redirected me to my search page. That was expected, but I noticed that elements that were supposed to be conditionally rendered only when a user was logged in were appearing in my navigation bar. This was causing myriad undesirable behaviors. What was interesting was that this only occurred upon refresh, if I simply clicked the icons to direct myself there without reloading the DOM, everything would appear normally.
• I discovered the back end was returning {'errors': ['Unauthorized']}. This was being registered as a truthy value by my front end and thus causing my navigation bar to display elements the user did not have the permission to see. I thought that changing the return value to None would rectify this bug; however, this triggered another bug:

      raise TypeError(
  TypeError: The view function for 'auth.authenticateSignup' did not return a valid response. 
  The function either returned None or ended without a return statement.

• Since I had to return a truthy value but also had to indicate to my front end if the user had the proper permissions, I changed my code back to the previously returned object. I then changed all the front end code relying on the presence of a user to check for !user.errors instead of user. This was partially effective but then I started getting Cannot read property of null errors in my front end.

Resolution

• I conditionally assigned a boolean to a variable based on whether the conditions were met to be considered a user (i.e., user object present and no errors coming from the back end).

   let isUser;
   if (user && !user.errors) isUser = true;
   else isUser = false;
  

  • I used this anywhere front-end user verification was needed and it was completely effective.

---

Time Toggle

Issue

• On the initial click of the element, there’s no change in state even though a dispatch occurs. Every subsequent click ostensibly behaves as desired.

Context

• I convert between 24-hour and 12-hour time at the click of an element on my History page. The conversion process is sound and the back end is being manipulated as expected.
• I pull the currently saved time preference from Redux store with useSelector() and set that as the default state for the time:

   useEffect(() => {
       	dispatch(readHistoryEntries());
       	dispatch(readUserSettings());
   	}, [dispatch]);
  
   const [toggledClock, toggleClock] = useState(clock24);

• The state is changed by the use of a toggle in the element’s click event:

   onClick={() => {
       toggleClock((prevClock) => !prevClock);
       editProfileHandler('clock_24');
   }}

• I couldn’t detect any issues with type or undefined values.

First Fix

• After reaching out, it was recommended that I change my state default for toggledClock from clock24 to !clock24. The fix was effective and consistent. It was suggested that this dissonance between dispatch and state could be attributed to the amount of values I’m changing.
  • This made me very curious. It made me think that perhaps I could leverage the asynchronism of async/await to create a more logical amendment if, indeed, it's the time disparity between data processes that is responsible for this.

Rumination

• Now, having had my logic reviewed and deemed sound. I set out to address this bug in a more semantic manner with a fresh perspective.
  • No matter what way I altered the code, however, I came back to square one until an idea dawned on me. Perhaps this was, indeed, a logical boolean scenario. It didn’t make sense that there would be timing issues since the dispatch was being hit immediately and there were no apparent race conditions.
  • I tried to reason with it and it gradually appeared to make sense. The toggled state must be initialized as the opposite of the currently stored preference. This is because all of the icons that represent conversion along with the alts and titles (which I also outfitted to inform the user dynamically), must be pointing to the what the state would be after toggling. For example, the 12-hour icon is presented to switch to 24-hour time and vice versa.
  • That’s the logic I had carried through the rest of the code in that section so it was reasonable that it applied here as well. What had confused me while hunting this error, though, is why it ever worked. If the initial condition was wrong, wouldn’t this result in an error consistently antithetical to the solution instead of working on the second click?
  • So, I attempted to trace the faulty logic even deeper and the more I dug into my code, the less it all made sense and the more I came to see that the previous fix was illogical. For instance, this line of code src={clock24 ? clock12Icon : clock24Icon}, which decides which image is displayed, doesn't make sense with the previous logic. If clock24 starts off truthy and is then negated with useState(!clock24); then how was the clock12Icon showing up?
  • There were many questions and odd behaviors but what embodies the issue is the following section in my edit profile handler:

    /* 
    With or without the bang in front of clock24, the
    toggle does not change the state of toggledClock
    within the handler. This is true even when it 
    outputs the desired boolean.
    */
    
    const editProfileHandler = (eType) => {
        if (eType === 'clock_24') {
            console.log(toggledClock); // prints true
            toggleClock((prevClock) => !prevClock);
            console.log(toggledClock); // still prints true
            dispatch(editProfile({
                clock_24: toggledClock,
                column: eType,
            }));
        }
    };   

    • The fact that the toggleClock was not toggling correctly brought me to realize that I would likely need to go back to the drawing board.

Resolution

• I began thinking that perhaps all of the state needed to be stripped down. All of the conflicts arising were likely due to a jumbled state. So, to address the degraded data flow, I boiled my code down to the basics. I brought in if/else statements and used a let variable initialized to clock24.

  let toggledClock = clock24;
  
  const editProfileHandler = (eType) => {
      if (eType === 'clock_24') {
          if (clock24 === true) {
              console.log(toggledClock); // prints false
              toggledClock = false;
              console.log(toggledClock); //prints true
          } else {
              console.log(toggledClock); // prints true
              toggledClock = true;
              console.log(toggledClock); // prints false
          }
          dispatch(editProfile({
              clock_24: toggledClock,
              column: eType,
          }));
      }
  };   

  • And just like that the code started behaving as desired. I was initially wary of variable reassignment since it often falls short of adequately managing state; however, this was definitely a successful use case.

---

Styling Create Theme Section

Issue

• I was running into issues styling my create theme form since I wanted to display horizontally in three rows. I was initially wondering how I would position 10 elements with their respective labels in three rows, dynamically changing based on the screen width.
  • There were other peculiarities like element borders misaligning because of their container's sizes and dynamic style changes for my theme tester; however the positioning of the elements was the crux.
  • Grid was an option; however, the labels would be displayed inline which threw off my hopes of having them above the inputs in their respective rows. So I though that absolutely positioned elements within an a relatively positioned parent would be the way to go.
    • That was partially successful but there were considerable obstacles one would have to traverse to make them truly dynamic, obstacles that would require the help of media queries.
    • I reconsidered a built-in solution, ruminating on how one could position those three rows of inputs with each label centered on top of its input.

Resolution

• After some planning, I came to the conclusion that a mix of flexbox, grid, absolute, and relative positioning would perfectly fit my needs. What I did was make a grid on the form level with three rows. Each row had display flex. Within each row were divs containing each input and its label and those divs also had display flex. I then positioned the buttons with position absolute in the relative container which held the form.
• It may initially seem excessive but it was super intuitive how it all came together. I had previously spend a tremendous amount of time trying to get the absolutely positioned elements to be completely dynamic and as soon as I switched my plan and used the grid-flex-flex strategy, my inputs were positioned correctly in minutes.

---

Mixing Data With Form Data

Issues and Resolutions

I1

werkzeug.exceptions.BadRequestKeyError: 400 Bad Request:
The browser (or proxy) sent a request that this server
could not understand. KeyError: 'setting_id'

R1

• I discovered that I was using request.files for non-file keys of the form data. I changed my program to accept the non-file data with request.form and the file data with request.files and everything started coming together.

I2

raise TypeError("Not a boolean value: %r" % value) 

R2

• I found that formData.append turns the appended data into a string which I had to account for on the receiving end (my Python routes) since my db was expecting a boolean.
  • Albeit the input data (checked attribute of a checkbox) came back as a boolean, it lost that type in the process of appending to form data and had to be type-casted back in my Python routes.
• This allowed me to phase out the previous system I had for uploading media and other theme data.
  • I was sending two requests for every theme card creation and edit. This was to capture all the style specifications as well as the file content.
  • The first request would get all the theme information besides media and the second request would be a patch to immediately update the db with the media data, giving the illusion of a singular request to the client.
  • Now all data could be appended to formData and retrieved on the Python routes side via request.form or request.files, depending on what was appropriate.

---

Flask Migration

Issue

File "/Users/.../Creepy_Crawler
/.venv/lib/python3.9/site-packages/flask_login/utils.py", line 272, in decorated_view
return func(*args, **kwargs)
TypeError: edit_user_profile() got an unexpected keyword argument 'settingID'

• There was no parameter in my function to match the route param. So, albeit the function param was not explicitly used within the function, it was still needed for the route.

  @user_routes.route('/profile/<int:settingID>', methods=['PATCH'])
  ...
  def edit_user_profile(settingID): 
  ... 

Resolution

• I encountered a situation where Flask-Migrate wasn't updating my tables. I had to and manually set the type of one of the erroring table columns to resolve the issue. This occurred after I had un-seeded, migrated, and upgraded several times for my project. The error would consistently show the wrong data type for one of the columns I had just updated. Later, I discovered that this (along with modifying the Alembic files directly) may have resulted in my database's corruption. I ended up having to drop and recreate my database locally and on Heroku. What I've learned is that individual changes to values and types within a given column don't reliably register in Alembic so it's best to just delete, migrate, upgrade, add the modified column, migrate, and upgrade if you want to see the changes to a section of your column reflected in your database.

---

404 on First Render

Issue

127.0.0.1 - - [09/Nov/2021 14:59:28] "GET /creepycrawler/settings/ HTTP/1.1" 200 -
127.0.0.1 - - [09/Nov/2021 14:59:45] "GET /settings HTTP/1.1" 404 -

First Fix

• This problem arose out of dissonance between the loaded assets from my Flask back end and my App.js routes. Everything would load, and the site would be fully functionally; however, the error would persist on the first render of each page except root and if I followed a link to an external URL and navigated back to my site with the history back arrow, it would raise a 404 error on the page. I changed my front-end routes to reflect my back-end routes and the issue was rectified. This meant changing the route in App.js from /settings to /api/settings/ and then changing any reference to it, such as nav bar button to the same route.
• After addressing this, I would get an authorization error in the console when accessing login and signup which could not be amended by simply making the Flask routes outfitted for both GET and POST requests. I discovered that I had an auth route in my back end for both my login and signup authorization which needed to be split up into a route for either (in my back-end auth routes and my front-end session store).

Resolution

• This was a false positive, the error does not appear on the deployed site. Changing the App.js and Flask routes to be the same results in the undesirable behaviors outlined in Browser Caching.

---

Handling Internal Server Errors

Issue

• I was getting CSRF Token expired after leaving my application open for the night then awaking and trying to submit a form. It was popping up as an alert which is how the errors that pass my first layer—pre-validation—get formatted. This was undesirable, but in order to do anything about it I'd have to see the error again.
  • So, I had to dig into how the token expiration was implemented in WTForms, and so I perused my .venv folder library and found the file responsible. I attempted to change "timedelta" to seconds in order to see it immediately; however, this was unsuccessful so I left it at minutes and changed the value to 1. This too was unsuccessful in triggering the error.
  • I shut down my back end and my front end and tried again, still wasn't triggered. I went into dev tools and attempted to change the expiration of the CSRF_TOKEN; however, a new one would just generate.

First Fix

• For time efficiency purposes, I did what I’ve done in other scenarios where testing a case-specific invalid value was not viable for a given validation: I plugged in an expression that would resolve to a falsy value like "1 = 2" in my back-end routes and changed the status code for the error to 500.
  • That way I could emulate what the response would be if the front end received a 500 status code error. I then wrote conditions to outfit all my front-end routes to catch such an error, refreshing the DOM if a back-end server error were to arise instead of delivering them a message to do it themselves or doing nothing at all.
  • The only thing I have to watch out for is the fact that this can mask errors that occur if one isn’t aware of or forgets this auto refresh. I’ve observed several internal server errors that would have escaped my attention had I not been vigilant. Notwithstanding, it seems to be a sound fallback for the client. This is natural, so often in the process of protecting the client or program, obstacles present themselves for testing or bug hunting. HTML pre-validations attributes are an example. One has to comment them out just to test one’s back-end forms. So, as with any codebase, one should be wary and willing to circumvent protective code for testing.

Resolution

• I later discovered, when working on my crawler, that the CSRF token was actually a 400 error with the exact message being The CSRF token has expired.. I left the previous code, which had become a nice fallback for 500 errors that might sneak into the client's view; however, I added a condition before my alerts to reload if data.errors[0] === 'The CSRF token has expired.' Note that I had to key into the data being retrieved since it is an array.

---

Object Not Iterable in Flask Form

Issue

TypeError: 'function' object is not iterable

Resolution

My Flask validation form had a custom function as the second argument (after the field label) in one of my input validations. That's why the program was rejecting it, it needed to be in a list and assigned to an attribute:

password = PasswordField('password', validators=[DataRequired(message='xyz'), customFunc])

not 

password = PasswordField('password', customFunc)

---

Field Manipulation in Flask Forms

Issue

I was wondering just what it would take to confuse my Flask forms since the labels I have in my back-end form validators do not match my front end, and I realized that WTForms may be resolving this by matching the input type with the field provided in the back end. So, I changed my string field to a password field for my email input and it didn't know where to look, even when the label was specified. My next test was to see if changing the order of same-type inputs could throw it off. I changed all references to the input with fruit names and only the variable names were left un-tampered with for later reassembly. The inputs were still registering correctly which really struck me as bizarre.

Finally, I changed the variable names too, making some comments to put it back together after. That broke it! It was really wild to see how the form was picking up on the variable names of each field and attempting to resolve that with the label coming from the front end. The only place I've encountered anything similar in my Finstagram project, I'll quote what I wrote about it here:

For security and aesthetic purposes, the autocomplete is turned off on
the forms. Google has pattern matching algorithms, however, which
attempt to pick up any indicators that the field is referencing an
email. It searches for placeholders and if no placeholder is set, it
looks at any available labels, in my case it tried to read my
aria-label. Note that autocomplete='off' doesn't work by itself though
it does take care of some inputs. One has to experiment with words to
bypass this behavior.

Resolution

This curiosity ended up paying dividends when I encountered a bug in another form. I couldn't get one of the validations to work.

I changed the variable in my form to match the name of the label on my front-end input and it worked liked a charm. I found that the letter case can be different and spaces on the front end are represented as underscores on the back end. This confirmed a few conjectures I had about how flask must be resolving the front-end label and back-end variable. Look at the following example which models what I gleaned:

• Back-End Form

    var_name = StringField('text', [Length(max=10, message='xyz')])

• Front-End Form

   <label htmlFor='x-y-z'>Var Name</label>

---

Back End Conditional Nesting

Issue

events.js:292
      throw er; // Unhandled 'error' event
      ^
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at writeAfterEnd (_http_outgoing.js:668:15)
    at ServerResponse.end (_http_outgoing.js:789:7)
    at ServerResponse.end (/Users/alejandrogrant/Desktop/code_playgrounds/app-acad-projects/key_projects_aa/Creepy_Crawler/react-app/node_modules/compression/index.js:107:21)
    at ProxyServer.<anonymous> (/Users/alejandrogrant/Desktop/code_playgrounds/app-acad-projects/key_projects_aa/Creepy_Crawler/react-app/node_modules/react-dev-utils/WebpackDevServerUtils.js:345:9)
    at ProxyServer.emit (/Users/alejandrogrant/Desktop/code_playgrounds/app-acad-projects/key_projects_aa/Creepy_Crawler/react-app/node_modules/eventemitter3/index.js:210:27)
    at ClientRequest.proxyError (/Users/alejandrogrant/Desktop/code_playgrounds/app-acad-projects/key_projects_aa/Creepy_Crawler/react-app/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:165:18)
    at ClientRequest.emit (events.js:315:20)
    at Socket.socketErrorListener (_http_client.js:469:9)
    at Socket.emit (events.js:315:20)
    at emitErrorNT (internal/streams/destroy.js:106:8)
Emitted 'error' event on ServerResponse instance at:
    at writeAfterEndNT (_http_outgoing.js:727:7)
    at processTicksAndRejections (internal/process/task_queues.js:81:21) {
  code: 'ERR_STREAM_WRITE_AFTER_END'
}

Resolution

This is the second time I ran into this error, last time was with AWS when I included application/json in the header for my file transfer to my back end (before I embedded the request in formData with other input values). It shut down my entire front end after being hit just like before; however, this time it was attributed to the structure of my back-end routes. I nested all the conditions in my edit route in a manner that was returning something unexpected to my front-end store. So, what I did was reorganize my back-end routes so that instead of nesting every condition inside of a higher order condition and returning errors if any of those internal conditions were not met, I paired each error return with its negative condition and returned if that negative condition was met. This solved the issue and made the code far clearer and open to maintenance.

For reference, instead of doing:

if a in b:
    do something = c
return { 'error': 'unsuccessful' }
    if c in d:
        do something = e
    return { 'error': 'unsuccessful' }

I did the following:

if a not in b:
    return { 'error': 'unsuccessful' }

do something = c
if c not in d:
    return { 'error': 'unsuccessful' }

---

JavaScript Global Flag Regex

Issue

• Not only does the JavaScript g flag not stop at the first match within a given string, it will match over subsequent calls.

Resolution

• To solve the bug I encountered, I stripped down my surrounding code. This allowed me to isolate and eliminate the behavior. I had multiple test methods being called for debugging purposes but they were ironically perpetuating the bug by summoning the g flag before it was intended. I removed the extra methods along with any unnecessary g flags. This is the behavior reduced to its most bare components:

      const r = /\w/g;
      const testA = r.test('word'); // true
      const testB = r.test('1');  // false

---

Heroku Deployment

Issue

• As I was cleaning up my directory structure, I noticed superfluous node_modules folders in my project root. After deleting them, other errors started to arise, locally and when trying to deploy. I tried several things but finally one error gave me steps to follow in my front-end console:

Resolution

To fix the dependency tree, try following the steps below in the exact order:

  1. Delete package-lock.json (not package.json!) and/or yarn.lock in your project folder.
  2. Delete node_modules in your project folder.
  3. Remove "eslint" from dependencies and/or devDependencies in the package.json file in your project folder.
  4. Run npm install or yarn, depending on the package manager you use.

Rumination

• Sometimes discrepancies between the Heroku database and your local database (for which your local environment is outfitted) can cause issues.
  • Using Heroku logs can assist in diagnosing what the error's cause. Here's how to assure Heroku's database is up-to-date.

    heroku run -a creepy-crawler-1 flask seed undo
    heroku run -a creepy-crawler-1 flask db migrate
    heroku run -a creepy-crawler-1 flask db upgrade
    heroku run -a creepy-crawler-1 flask seed all

  • Dissonance between Redux routes and back-end routes can also cause issues, such as data not displaying on the deployed front end albeit some of the site may be visible and even functional. I've observed that trailing slash differences can be a cause of this and can be a false negative on localhost.
  • For GitHub action's auto deployment, you may run into obstacles if you don't have the HEROKU_API_KEY in your repository's Secrets section or if you're missing the YAML code block.
    • to generate a HEROKU_API_KEY, copy the token after entering heroku authorizations:create into the terminal
    • this is the YAML code block once you choose the option to set up a workflow yourself:

      name: Python application
      on:
      push:
          branches:
          - main
      jobs:
      build:
          runs-on: ubuntu-latest
          env:
          HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }}
          steps:
          - uses: actions/checkout@v2
          - name: Login to Heroku Container registry
          run: heroku container:login
          - name: Build and push
          run: heroku container:push -a creepy-crawler-1 web
          - name: Release
          run: heroku container:release -a creepy-crawler-1 web
      

---

Getting ESLint to Work

Issue

• How do I get ESlint to run in my chosen directory?
  • I did npm install --save-dev eslint and npx eslint --init, then configured the eslintrc.json in my root but all I have are the warnings that come with the VSCode ESlint extension. The rules I set up in that eslintrc.json only work in my top-most directory, not my react-app.

Resolution

• I deleted all package.json files in my root (they had only been created in a previous unsuccessful attempt to get it to work in all directories and were unnecessary bloat), solely keeping the extension on my IDE (VSCode) and then I made a file in my react-app directory called .eslintrc.json. Next I placed the following code and the linter immediately picked it up and started abiding by the style guide I picked:

  {
    "extends": ["airbnb-base"]
  }

• One must assure that the install commands are run on the first level of the react-app (front-end) directory.
• npx install-peerdeps --dev eslint-config-airbnb will solve the error that states airbnb-base can't be extended to.
• Deleting "parser": "babel-eslint", from the .eslintrc.json file will solve the error that mentions babel-eslint-parser.

---

History Entry Order

CRUD — create, read, update, delete

back-end routes

At this point, my object is this in the back end:

It's a list of 87 entries ordered how I would like. It is then parsed by json on the front end here:

The order and functional structure is maintained, the only visible difference being that the indices are visible in the front-end console:

Where the data finally loses its order is in the READ_HISTORY case of my history reducer:

The data is now ordered by index:

It will also appear as such in my state.

The conundrum is the following: I need to have my indices match the ids for the sake of O(1) lookup time for my patch and delete operations. So, albeit I have been able to maintain the order of the data by spreading entries and newState in my READ_HISTORY case: return {...entries,...newState} instead of using a forEach method, the result is only beneficial for my read operation.

As you'll see in the following picture, the re-indexing kills any instant lookup time chances:

What I wanted:

• O(1) lookup time for UD operations
• custom ordered back-end data
• minimal dispatches

Thought I could:

• eliminate the delete and update cases from my switch (they won’t be altering the front end directly)
• spread my entries into new state (to preserve the back-end ordered data)
• trigger refresh upon after redirecting to target page

Issue that arose:

• lost all seamless rendering, had to rely heavily on refresh to update state

First Fix

• I made a cached reference object in history_store.js which finally gave a key reference for my UD operations, allowing for O(1) lookup.
  • It prevents me from having to refresh my entire DOM every time an entry is deleted
  • As mentioned, my update operation can seamlessly change a clicked element's data but the order of all the elements on the DOM can only be changed on refresh since that information is coming from the thunk belonging to the read operation.
  • In this instance, the O(1) lookup time is overshadowed by the fact that I refresh upon redirect, so by the time one goes back to history, everything is ordered properly.

Resolution

• I made my back end return more. Instead of creating a reference object on the front end to manipulate the DOM then refreshing anytime the history page was accessed to prevent from duplicate key errors in React, I just dispatched the newly ordered content for each CRUD operation.

• The database is already being manipulated, all that was added was a more general query to grab all of the entries afterward and a return that type-casted the all the ordered entries into a dictionary as opposed to merely a success message (for DELETE) or an individual entry (for UPDATE and CREATE). READ was already outfitted for this activity. This eliminated the following:

  • all window refreshed when accessing history
  • reference object for CRUD operations
  • entry data updates but history does not reorganize

• The previous way was giving the illusion of a dynamic page by manipulating the DOM directly and refreshing it right before it was re-accessed (e.g., by a search entry being added or by clicking to another page on the site and clicking back). This destroyed any performance benefit the reference object offered and detracted from the UX.

• In conclusion, the solution to my initial conundrum was to manipulate my back end as usual (e.g., send delete request, have db delete) but return a more thorough response to my front-end (i.e., all entries instead of merely one entry or a success message) for dispatch.

  • One can see how substantial this development was for my application as it eliminated the last of page refreshes governing my site's core functionality (only present for 500 internal server errors now)—a true boon to my application's performance, the user experience, and code readability.

For reference, my history reducer went from:

const initialState = {}
let newState;
let historyCache = {};

export const historyReducer = (state = initialState, action) => {
    newState = {...state};
    switch (action.type) {
        case CREATE_HISTORY:
            const entry = action.payload.history;
            newState[entry.id] = entry;
            return newState;
        case READ_HISTORY:
            const entries = action.payload.history;
            entries.forEach((entry, idx) => historyCache[entry.id] = idx)
            return {...entries,...newState};
        case UPDATE_HISTORY:
            const updateEntry = action.payload.history;
            newState[historyCache[updateEntry.id]].updated_at = updateEntry['updated_at'];
            newState[historyCache[updateEntry.id]].tz = updateEntry.tz;
            newState[historyCache[updateEntry.id]].tz_abbrev = updateEntry['tz_abbrev']
            return newState;
        case DELETE_HISTORY:
            const entryID = action.payload;
            delete newState[historyCache[entryID]];
            return newState;
        default:
            return state;
    }
}

to the following:

const initialState = {};

export const historyReducer = (state = initialState, action) => {
    switch (action.type) {
    case CRUD_HISTORY: {
        const entries = action.payload.history;
        return { ...entries };
    }
    default:
        return state;
    }
};

---

Back-End History Entry

Error Type: Semantic

Issue

My search is not being logged to my database's history table.

Context

Front End

The entry is made front_end_display

I send the relevant data from my search form's onChange event and dispatch in my searchHandler function front_end_search_page_component

It's sent to my store and processed by my thunk and reducer front_end_history_thunk

front_end_history_reducer

By this point, I'm receiving all the values I expect. Initially, I faced an issue with new Date() unexpectedly returning an object albeit appearing as a string; however, that was rectified. Here is what my console looks like:

front_end_console

Back End

Bug Manifestation This route is hit:

back_end_history_routes

The print statement doesn't make it inside of the conditional, as evinced in my terminal:

back_end_terminal
This makes me think that the form may not be validating as expected.

Possible Bug Origin The route summons the following form:

back_end_search_form

Here are relevant snippets from my database tables:

back_end_histories_table

back_end_users_table

Resolution

• csrf token in routes
  • or can't post
• trailing slash in store request
  • prevented need to use strict_slashes=False to address Flask trailing slash error
  • note that trailing slashes are used for signifying groups of data, whether referencing a directory in a file structure or a collection hit by a route, a trailing slash is the semantic way to go
• keys dispatched in search page must match back_end route and form keys
  • or can't post
• headers in post
  • or can't post
• visit needs to allow null
  • or can't post

---

AWS Integration

Issues and Resolutions

Q1

• Logging formData directly prints an empty object. How do I check if I appended to formData correctly?

A1

const formData = new FormData();
Object.fromEntries(formData.entries());

Q2

• My request is dying between my front end and back end. How do I rectify the following errors, the latter which shuts down my entire front-end server?

Could not proxy request /api/users/2 from localhost:3000 to http://localhost:5000

Error [ERR_STREAM_WRITE_AFTER_END]: write after end 

A2

/* 
Remove this from your requesting thunk.
The form data has its content type
(e.g., image/jpg) so this throws it off.
*/
headers: {
    'Content-Type': 'application/json'
},

Q3

• How do I fix the error hex attribute doesn't exist on uuid4?

A3

• In s3_helpers.py, invoke uuid4 (i.e., uuid.uuid4().hex).

Q4

• How do I fix the following error?

An error occurred (InvalidAccessKeyId) when calling the PutObject
operation: The AWS Access Key Id you provided does not exist in
our records.

A4

• Change Block public access (bucket settings) in Permissions section of s3 bucket to the following.

   Block all public access
   	Off
   Block public access to buckets and objects granted through new access control lists (ACLs)
   	Off
   Block public access to buckets and objects granted through any access control lists (ACLs)
   	Off
   Block public access to buckets and objects granted through new public bucket or access point policies
   	Off
   Block public and cross-account access to buckets and objects through any public bucket or access point policies
   	Off
  

• Change Bucket policy settings, also located in Permissions section of s3 bucket to the following.

   {
   	"Version": "2012-10-17",
   	"Statement": [
   		{
   			"Sid": "Stmt1420751757000",
   			"Effect": "Allow",
   			"Action": "s3:*",
   			"Resource": "arn:aws:s3:::NAME_OF_BUCKET/*"
   		}
   	]
   }

Rumination

• In order to immediately identify the origin of the bug (ie., aws account settings or my code), I uploaded manually to my bucket on my browser and tried the view the image url. It gave back a concise error telling me I wasn't authorized to view the upload.

   <Error>
   <Code>AccessDenied</Code>
   <Message>Access Denied</Message>
   <RequestId>7WDFCD7875TP6ABH</RequestId>
   <HostId>G9bv6r48cWIZhHj3JL9hozikFIm/V/0q/Z8nqrNNtgnF1jplgIk+KEtcMn9yBJkzQgN1ro99qYw=</HostId>
   </Error>
  

• Initially, I wasn't able to have latest credentials recognized. I had changed my credentials several times for various reasons and when I wrote the following debugging code into my program, it evinced that aws wasn't getting my most recent access key ID.

   def upload_media(userID)
   	import boto3
   	print(boto3.set_stream_logger('botocore', level='DEBUG'))

  • The print was a comprehensive debug log with one part that particularly stood out:

     <!-- This did not match my most current access key ID -->
     </Message><AWSAccessKeyId>12345EXAMPLE</AWSAccessKeyId>

  • I shut down all of my servers and restarted my shell. This addressed the previous error, but that's when I first encountered the <Message>Access Denied</Message> error. This would foreshadow what I would later view on my aws account when I uploaded manually. As aforementioned, this was rectified by changing my Bucket policy settings.

Q5

• How do I solve this error: AccessControlListNotSupported: The bucket does not allow ACLs?

A5

• Buckets > Permissions > Edit Object Ownership > Select ACLs enabled

---

Unexpected End of JSON Input Port Issue

Issue

I keep getting the issue unexpected end of JSON input. It also doesn't appear like my Flask endpoint is being hit since print statements are not working.

Resolution

I had changed several locations that either reference or influence the port that Flask uses, such as:

• internal_port altered in fly.toml
• FLASK_RUN_PORT altered in .flaskenv
• EXPOSE altered in Dockerfile

However, I had forgotten to change the critical connection point between the front and back end:

• proxy altered in package.json

---

Memory on Deployed Site Running Out

Issue

On fly.io, the search engine kept timing out no matter how long I allowed the front end to run the query before refreshing. Upon looking at the logs from fly.io I noticed the following error: Out of memory: Killed process 519 (gunicorn).

Resolution

• In the react-app directory, edit the scripts object in the package.json so that "start": "react-scripts start" is changed to "start": "cross-env NODE_OPTIONS=--max_old_space=192 webpack". This will free up more memory and prevent the application from crashing on each query on the deployed site.

---