feat(map): cover map with HTMLElements

[DwieDima]

Is your feature request related to a problem? Please describe.

Currently it's not possible to implement an application using sidemenu or modals while the map is active in background.
Therefore you can't implement many use cases that you find in many apps (compare various scooter sharing apps etc.)

Here you can see a sample application which is using sidemenu and modals with the latest version of capacitor-google-maps plugin

Describe the solution you'd like

The map should be able to run in the background with HTML elements overlapping (sidemenu, modals, drawer...)

Additional context

here a some reference applications with complex UI covering the map (LiveParking is a ionic application using google maps cordova).

LiveParking	Pick-e-Bike	Bird

I am aware that it is not an easy task, you also mentioned that there is currently no solution for it.
Nevertheless, I would like to place the feature request here.

this feature request is related to #18 #14

[hemangsk]

Thanks for creating this, will be easier to document all the progress/blockers on this feature here.

I'm aware of an embarrassing workaround for this, that we can hide and show the map with CapacitorGoogleMaps.show(), CapacitorGoogleMaps.hide() by listening to the modal events 🤦

This hack kind of works for the side menu use-cases for small apps, but that's where it is limited to.

[DwieDima]

Thanks for creating this, will be easier to document all the progress/blockers on this feature here.

I'm aware of an embarrassing workaround for this, that we can hide and show the map with CapacitorGoogleMaps.show(), CapacitorGoogleMaps.hide() by listening to the modal events 🤦

This hack kind of works for the side menu use-cases for small apps, but that's where it is limited to.

closing and showing the map would, as you said, lead to a poor user experience with larger apps, as you would expect to see parts of the map anyway.

I would like to thank you for working on this plugin. I'm always happy to see your plugin at the top of the capacitor community repo (so there were new updates pushed 😄).

[tafelnl]

I think it would be wise to close the other issues in favor of this one @hemangsk :)

[tafelnl]

As far as I understand cordova google maps solves this by rendering the MapView behind the WebView. The WebView should then be made transparent in order to show the map.

But because of that, another problem rises. The MapView can now not be touched anymore because all the touch events are delegated to the overlapping WebView of course.
Cordova google maps solves that problem by detecting what element the user touches. If it detects the user taps HTML elements it will delegate the touch to the WebView and if it detects the user taps the map it will delegate the touch events to the MapView.

But I honestly have no idea of how to accomplish this. Anyone got an idea?

[tafelnl]

Related SO questions:

• https://stackoverflow.com/questions/48269124/create-a-hole-in-react-native-webview-for-touch-events
• https://stackoverflow.com/questions/64443259/android-webview-overlay-with-clickable-and-transparent-parts
• https://stackoverflow.com/questions/19676712/pass-touches-to-the-view-under

[hemangsk]

https://github.com/capacitor-community/camera-preview has a toBack option which looks like what we need here as well 🤔

[tafelnl]

@hemangsk Yes that's exactly what I mean. It is not even that hard to implement. I already did it with capacitor-community/barcode-scanner. But we will have to tackle to problems I mentioned before first. Because otherwise the whole map will be unusable because it will not register any touch events.

[hemangsk]

Ah okay! lets figure out a solution for click events delegation. going through the resources linked SO resources

[DwieDima]

here i found maybe some useful links for touch event propagation

ios

• https://stackoverflow.com/a/33819119/15999857
• https://www.tutorialfor.com/questions-110880.htm
• https://stackoverflow.com/questions/61208881/cant-handle-touches-simultaneously-with-wkwebview-on-ios-13-4
• https://www.programmersought.com/article/54855839772/
• https://greensock.com/forums/topic/21397-ios-13-safariwkwebview-draggable-double-dispatching-click-events/
• ios 13 safari release:
  • "Added support for the Pointer Events API enabling consistent access to mouse, trackpad, touch, and Apple Pencil events."
  • https://developer.apple.com/documentation/safari-release-notes/safari-13-release-notes

android

• https://programmersought.com/article/55002145124/
• https://stackoverflow.com/questions/57159810/how-to-get-touch-event-information-in-android-webview
• https://stackoverflow.com/questions/65663648/how-to-stop-touch-events-in-android-webview-when-its-content-is-scrolling
• disallow touch events for google maps when propagating webview events (so it won't trigger twice)
  • https://developer.android.com/reference/android/view/ViewParent#requestDisallowInterceptTouchEvent(boolean)
  • https://gist.github.com/nuthatch/1747558
• translate camera movement from webview
  • (get x y coord from webview and translate to lat/lng) https://stackoverflow.com/questions/52996302/modify-coordinates-with-pixels-in-android-google-maps
  • (move camera to lat/lng) https://stackoverflow.com/a/28129455/15999857

[tafelnl]

Just food for thought:

Maybe we can do something like this:

• Set a touch listener on the <html> element and also one on the element that the map is attached (lets call it <mapdiv> for now) to.
• Then on ontouchstart event, notify the native API that all touch events should be delegated to the MapView from now
• Finally on ontouchend event, notify the native API that all touch events can be delegated to the WebView again.
• We should also check that the touch events really target the <html> and <mapdiv> and not some overlaying other div. This can be done with (Touch.target](https://developer.mozilla.org/en-US/docs/Web/API/Touch/target) I think.

I have no idea if what it does to the performance or if it would work in the first place, but I guess we can just test it out./

Unfortunately, at the moment, I do not have time for this. So if anyone would like to volunteer to test out this idea. It would be more than welcome :)

Edit: On closer inspection, I think that only adding a listener to <mapdiv> (instead of both <html> and <mapdiv>) should do the trick already.

[DwieDima]

Just food for thought:

Maybe we can do something like this:

• Set a touch listener on the <html> element and also one on the element that the map is attached (lets call it <mapdiv> for now) to.
• Then on ontouchstart event, notify the native API that all touch events should be delegated to the MapView from now
• Finally on ontouchend event, notify the native API that all touch events can be delegated to the WebView again.
• We should also check that the touch events really target the <html> and <mapdiv> and not some overlaying other div. This can be done with (Touch.target](https://developer.mozilla.org/en-US/docs/Web/API/Touch/target) I think.

I have no idea if what it does to the performance or if it would work in the first place, but I guess we can just test it out./

Unfortunately, at the moment, I do not have time for this. So if anyone would like to volunteer to test out this idea. It would be more than welcome :)

Edit: On closer inspection, I think that only adding a listener to <mapdiv> (instead of both <html> and <mapdiv>) should do the trick already.

As alternative you can intercept all touch events on the native side. So on web you just declare the div element which should be intercepted. There are API's on ios and android side

• for ios there are UIGestureRecognizers
• for android there are MotionEvents

I referenced some examples here

Since i'm not a ios/android developer this is also some food for thoughts and needs investigation.

[tafelnl]

Yes that could also be used maybe. But the thing with that is: how would you know if the user tapped on an overlay element or on the map?

[DwieDima]

Yes that could also be used maybe. But the thing with that is: how would you know if the user tapped on an overlay element or on the map?

you would never actively click on the map with this approach, but would have to programmatically transfer each event (click, swipe, pinch, doubleclick ...) to the map.

I can imagine it like this:

• (web) user clicks on div (where map is defined)
• (native) touchEvent interceptor receive an click event with x and y coord
• (native) programmatically click the map on the x y coord
• (native) map shows the result to user

this is where it gets interesting when it comes to pinch gestures.

[AE1NS]

There should also be a listener of parent scroll events to dynamically change the map offset on page scroll. I.e. on init there could be a check starting from the map parent container up to the document and checking each container for something like:

const isScrollable = elem.scrollHeight > elem.clientHeight || elem.scrollWidth > elem.clientWidth;
elem.addEventListener("scroll", function() {
    // Recalculate map offset
});

[AE1NS]

Yes that could also be used maybe. But the thing with that is: how would you know if the user tapped on an overlay element or on the map?

Only one native listener cant be the solution. As you mentioned, the semantic cases of the user interaction with different overlays inside the webview must always be checked in parallel. Any user interaction on the map div itself inside the webview must release/activate or lock the native listener.

As alternative you can intercept all touch events on the native side.

But is it useful to use the native listener if the webview events always needed to be tracked too, or is it a better approach to pass the javascript events to the native code? Performance etc.?

[DwieDima]

Just food for thought:

Maybe we can do something like this:

• Set a touch listener on the <html> element and also one on the element that the map is attached (lets call it <mapdiv> for now) to.
• Then on ontouchstart event, notify the native API that all touch events should be delegated to the MapView from now
• Finally on ontouchend event, notify the native API that all touch events can be delegated to the WebView again.
• We should also check that the touch events really target the <html> and <mapdiv> and not some overlaying other div. This can be done with (Touch.target](https://developer.mozilla.org/en-US/docs/Web/API/Touch/target) I think.

I have no idea if what it does to the performance or if it would work in the first place, but I guess we can just test it out./

Unfortunately, at the moment, I do not have time for this. So if anyone would like to volunteer to test out this idea. It would be more than welcome :)

Edit: On closer inspection, I think that only adding a listener to <mapdiv> (instead of both <html> and <mapdiv>) should do the trick already.

found a useful link to handle pointer events, including pinch gestures on javascript side

• https://developer.mozilla.org/en-US/docs/Web/API/Pointer_events/Pinch_zoom_gestures

[tafelnl]

Amazing news guys: I found a way to make this work on Android. It is actually very fast, efficient and reliable as far as I could figure out. Not sure if the method I used can also work on iOS, but I guess we will find that out soon enough. Will share a working example as soon as I finish some last tests!

[tafelnl]

As mentioned in #85 this plugin is going to be refactored altogether. As I have already made quite some progress (mainly Android), I implemented this functionality on top of that rewrite. Progress can be seen here: https://github.com/DutchConcepts/capacitor-googlemaps-native/tree/next

I have also just created a new repo with examples on how to use it.

Please check it out and share your feedback.

Unfortunately I do not have the time to implement this functionality for iOS. I might do in a month or so. But I think the concept should be clear and it shouldn't be to hard to implement it on iOS for someone else if anyone has time available.

[hemangsk]

@tafelnl This look great! 🚀 I'll also test it over the weekend and get back, it looks way better than the existing plugin already.

[leobaccili]

Very nice plugin, i'm waiting for this feature

[ludufre]

As mentioned in #85 this plugin is going to be refactored altogether. As I have already made quite some progress (mainly Android), I implemented this functionality on top of that rewrite. Progress can be seen here: https://github.com/DutchConcepts/capacitor-googlemaps-native/tree/next

I have also just created a new repo with examples on how to use it.

Please check it out and share your feedback.

Unfortunately I do not have the time to implement this functionality for iOS. I might do in a month or so. But I think the concept should be clear and it shouldn't be to hard to implement it on iOS for someone else if anyone has time available.

Hi, I'm working on your fork on the iOS version. Also, I implemented a method to get VisibleRegion and already created PR for you.

[selected-pixel-jameson]

@ludufre Per my conversations with @tafelnl the iOS implementation should be the last part that needs to get taken care of for that repo to be ready and it should have almost all of these features implemented. I contracted @tafelnl out to build most of that but he's gotten busy and has been unable to wrap up the iOS features.

[selected-pixel-jameson]

@ludufre Have you had any luck getting the iOS version working on your fork?

[ludufre]

@selected-pixel-jameson unfortunately no. I give up and migrated to the slow javascript sdk...

[selected-pixel-jameson]

I have figured out how to allow map elements to overlap the map in iOS. It comes with a few caveats though and I wanted to run it by the community.

In order to allow for the elements defined within the Webview to overlap the MapView the WebView needs to be placed above the map view in the iOS ViewControllers View Hierarchy.

In the capacitor framework documentation they give you access to self.bridge?.viewController and self.bridge?.webView.
In order to listen to touch events I had to make sure that both the MapView and the WebView were subviews of a custom parent view.

let passThroughView = PassThroughView(frame:(self.bridge?.viewController?.view.frame)!)

self.bridge?.viewController?.view = passThroughView

passThroughView.insertSubview(self.bridge?.webView as! UIView, at: 0)
passThroughView.insertSubview(customMapView.view, at: 0)

// This is a reference that is passed in in the initialization by looping through all of the child elements in the html container.
passThroughView.setInnerElements(elements: innerElements)

Then within the custom parent view I had to override the hitTest function and test to see if the point that gets passed is within the GSMapView and userInteractioinEnabled is true then return that subview, otherwise just proceed as normal. Furthermore I'm grabbing all of the bounds of child elements within the specified map element on the ionic side of things and testing to see if the point is within any of those. This allows buttons to be laid over the top of the map.

// Javascript Initialization
await CapacitorGoogleMaps.initialize({
        key: 'XXXXXXXXX',
        devicePixelRatio: window.devicePixelRatio,
      });
      
      // Get reference to element for coordinates
      const element = document.getElementById('map')
      const boundingRect = element.getBoundingClientRect();
      
      // Loop through all children of element to get their bounds
      let childElements = []
      element.querySelectorAll("*").forEach(child =>{
        if(child){
          childElements.push(child.getBoundingClientRect())
        }
        
      })
      
      // Create Map
      try {
        const result = await CapacitorGoogleMaps.createMap({
          boundingRect: {
            width: Math.round(boundingRect.width),
            height: Math.round(boundingRect.height),
            x: Math.round(boundingRect.x),
            y: Math.round(boundingRect.y),
          },
          cameraPosition: {
            target: {
              latitude: -33.86,
              longitude: 151.2,
            },
            zoom: 12,
          },
          preferences: {
            gestures: {
              isScrollAllowedDuringRotateOrZoom: false,
            },
            appearance: {
              style: refName === 'first' ? this.style : null,
              isMyLocationDotShown: true,
            },
          },
          innerElements:JSON.parse(JSON.stringify(childElements))
        });

override func hitTest(_ point: CGPoint, with event: UIEvent?) -> UIView? {
      for subview in subviews as [UIView] {
          if subview.isUserInteractionEnabled && subview.point(inside: convert(point, to: subview), with: event){
              let type = NSStringFromClass(type(of:subview))
              
              for rect in self.innerElements {
                  if rect.contains(point){
                      return super.hitTest(point, with: event)
                  }
              }
              
              if(type == "GMSMapView"){
                  return subview.subviews.first
              }
          }
      }
      return super.hitTest(point, with: event)
    
 }
 
 func setInnerElements(elements:JSArray){
        
        self.innerElements = []
        elements.forEach { element in
            if let jsonResult = element as? Dictionary<String, AnyObject> {
                let x = jsonResult["x"] as? Double ?? 0;
                let y = jsonResult["y"] as? Double ?? 0;
                let width = jsonResult["width"] as? Double ?? 0;
                let height = jsonResult["height"] as? Double ?? 0;
                
                let point = CGPoint(x: x, y: y);
                let size = CGSize(width: width, height: height);
                
                self.innerElements.insert(CGRect(origin: point, size: size), at: 0)
            }
        }
    }
 

Lastly, I had to implement calls that enable and disable the userInteractionEnabled flag on the mapView to allow for WebView interaction when we don't want the mapView to receive events. For example when we have a ion-menu that pulls out over the map we will need to call something like this

CapacitorGoogleMaps.disableMap({mapId:XXXXXXXX})

Then when the Menu is hidden we will have to listen for it using (ionDidClose)="enableMap()" and call CapacitorGoogleMaps.enableMap({mapId:XXXXXXXX})

The biggest caveat right now is dynamically sized elements that layover the map (i.e. ion-fab). The initial button will work, but the dynamic buttons that show up after you click will not be clickable. This is because the bounds of those elements has changed. While we could update them when the button is clicked I'm not sure how we would reset them back. Right now enabling and disabling the map would probably be an easier solution.

I know this is a lot. I'm working on getting this all polished and will get a repo out there for people to test with as soon as I can.

[va2ron1]

@selected-pixel-jameson just for reference, I have a similar solution but is for my needs. Maybe it can help to do a better solution. Also, I used this with the main package, not with the DutchConcepts fork. But it should work in both packages.

In the same Plugin.swift file after the imports I added:

class CustomMapView: UIView {
    var mapView: UIView!
    var webView: UIView!
    override func hitTest(_ point: CGPoint, with event: UIEvent?) -> UIView? {
        if isClear(point: self.convert(point, to: self.webView), layer: self.webView!.layer) {
            return self.mapView.hitTest(point, with: event)
        }
    
        return self.webView.hitTest(point, with: event)
    }

    func isClear(point: CGPoint, layer: CALayer) -> Bool {
        var pixel: [UInt8] = [0, 0, 0, 0]
        let colourSpace = CGColorSpaceCreateDeviceRGB()
        let alphaInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedLast.rawValue)
        let context = CGContext(data: &pixel, width: 1, height: 1, bitsPerComponent: 8, bytesPerRow: 4, space: colourSpace, bitmapInfo: alphaInfo.rawValue)

        context?.translateBy(x: -point.x, y: -point.y)

        layer.render(in: context!)

        return CGFloat(pixel[0]) == 0
            && CGFloat(pixel[1]) == 0
            && CGFloat(pixel[2]) == 0
            && CGFloat(pixel[3]) == 0
    }
}

And I replaced in the create method this line with the next code:

let customMapView = CustomMapView()

self.bridge?.viewController?.view = customMapView

customMapView.webView = self.bridge?.webView
customMapView.mapView = self.mapViewController.view
customMapView.addSubview(self.mapViewController.view)
customMapView.addSubview(self.bridge?.webView as! UIView)

self.bridge?.webView?.evaluateJavaScript("document.head.insertAdjacentHTML(\"beforeend\", `<style>ion-content { --background: transparent; }</style>`)", completionHandler: nil)

self.bridge?.webView?.isOpaque = false
self.bridge?.webView?.backgroundColor = .clear
self.bridge?.webView?.scrollView.backgroundColor = .clear

Using the main example, this how the HTML element should looks like:

<div id="map" #map>
    <ion-button expand="block">Overlay Button</ion-button> 
</div>

[tafelnl]

Closing this issue, since we have a solution now, which is merged into the next branch in #144. Major thanks to @va2ron1 for contributing