Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split US #143

Closed
3 tasks done
corradio opened this issue Nov 23, 2016 · 58 comments
Closed
3 tasks done

Split US #143

corradio opened this issue Nov 23, 2016 · 58 comments

Comments

@corradio
Copy link
Member

corradio commented Nov 23, 2016

@martindaniel4
Copy link
Contributor

👌

@sYnfo
Copy link

sYnfo commented Dec 31, 2016

Is this something that someone new to the project could help with? If so, could you please specify in more detail what would be needed to get US on the map?

@corradio
Copy link
Member Author

corradio commented Dec 31, 2016 via email

@corradio
Copy link
Member Author

@sYnfo are you still up for helping?

@corradio corradio self-assigned this Jan 12, 2017
@corradio
Copy link
Member Author

I started working on this at https://github.com/corradio/electricitymap/tree/data/us

@garciaev
Copy link

Hi -

I am interested in getting the USA on the map. Where do I start?

@corradio
Copy link
Member Author

Hi @garciaev ,

Thanks! First, we need to get data for the boundaries of each ISO, for which we already have production data. Can you help? I didn't succeed in finding geoJSON or shapefiles..

@garciaev
Copy link

OK just to be sure, we basically need geoJSON or shape files that have boundaries for the American ISOs here: http://www.isorto.org/about/default

Correct? I am currently searching for the California ISO boundary map.

@corradio
Copy link
Member Author

corradio commented Feb 15, 2017 via email

@jacobbaratta
Copy link
Contributor

http://www.eia.gov/beta/realtime_grid/ has two vector graphics that may be convertible with some work. I ripped the simpler vector graphic into a standalone svg, but I have zero experience with GIS and my short foray into investigating the feasibility of converting the SVG indicated that while it's probably possible, it's too complicated for a complete novice like me to attempt. I've attached the SVG to this post. Even if it can't be converted it can still probably help in some way with construction of a North American map.

eia-mod.svg.zip

@garciaev
Copy link

Hmm... not sure those EIA regions are the same as ISO member boundaries here? http://www.isorto.org/about/default

I think I am just gonna call the customer service and see if they can forward me to someone who can provide us with the boundary files.

@jacobbaratta
Copy link
Contributor

Hmm, apparently part of the issue is that the EIA calculates their regions slightly differently from the ISOs, leading to excessive fuzziness. http://www.eia.gov/beta/realtime_grid/#/status makes this clearer (in the site but not so much the graphic). I wonder if we could just pull the data from the EIA API and use their borders? It would probably provide superior coverage of the US, although it likely lacks some of the granularity of ISO data (you might be able to get the import/export data though?). Unfortunately the EIA's data for MISO mirrors MISO's data, so that does need to be compensated for (since MISO crosses the border into Canada to cover Manitoba)...

I fooled around with qGIS and got the first SVG file imported and aligned, but I realized that the resulting raster file isn't what's wanted...and it still lacks Canada entirely and still has the unavoidable misalignment problem... Even if I get the raster converted into a shapefile it'll still have the CA/misalignment issue, so I dropped that again for now.

IMO it might actually be easier at this point to just make a FOIA request for the data the EIA uses (since it doesn't seem to be available on their site). But that still leaves excessive fuzziness with the borders, and the issue of handling MISO in Manitoba, plus AESO and IESO. Theoretically those three independent regions are literally each divided solely along provincial/national borders, so you could probably just get border data elsewhere and call it a day, but you still have to merge the MISO Manitoba and MISO US regions.

@jacobbaratta
Copy link
Contributor

jacobbaratta commented Feb 20, 2017

I have crude (some annoying alignment issues with borders that can be obscured via thick borders) shapefiles (as individual outlines of regions) for MISO (this one is technically two shapefiles and I followed the coverage indicated on the MISO site for the Manitoba MISO shapefile, which is apparently only a small part of Manitoba rather than the entire province (not sure why but I'm not arguing with MISO's official map)), SPP, PJM, NYISO, and ISO-NE. I also have ERCOT and CAISO files, although the ERCOT file might not be up to snuff (alignment issues - but I can replace it with my alternate source that aligns better). I also found a shapefile with all North American electrical border crossings, not sure if that's useful but I left them in the preview below. AESO and IESO regions do not have shapefiles yet and I don't feel like creating them at this time (doing it manually is a pain, haven't found a viable way to do it automatically yet). There were no other major North American ISOs I could cover, so there are large coverage gaps.

Here's what it currently looks like zoomed out:

qgis screenshot overview2

Should I even bother continuing to clean this up and finish it or is the misalignment too bad to use? Ignore any misalignment alongside the ERCOT border and keep in mind that I set border line widths this thick to obscure misalignment on other borders (depending on zoom levels line widths might be reducible a bit further but not by much). All the data is georeferenced vectors saved as shapefiles.

@corradio
Copy link
Member Author

Hi @jacobbaratta,

Sorry for my late reply, and thanks so much for the work here. It is very appreciated.
I've been fooling around with those things myself, and I'd say that any solution that approximates reality well enough is good to start with.
Basically, here's the ISOs for which we have production data for, and for which we need the corresponding boundaries: https://pyiso.readthedocs.io/en/latest/options.html
The script that generates the boundaries for Europe (and the US) can be accessed here: https://github.com/corradio/electricitymap/blob/master/web/topogen.sh
It uses shapefiles, so if we can generate shapefiles for the different ISOs required, we should be able to merge them with the existing US border given in the script (and retrieved from www.naturalearthdata.com).

Does that make sense? I believe a simple solution to start with is always good enough. We can always improve later. The important part is to get started!

@jacobbaratta
Copy link
Contributor

jacobbaratta commented Mar 1, 2017

Okay so I need to redo the ERCOT border first and foremost...

I looked into the feasibility of adding BPA and/or NVEnergy since they are on the pyiso list. Unfortunately there is no good map for BPA (maps exist but they're too fragmented to use), so that will need to be shelved indefinitely.

NVEnergy appears to be generalizable to the whole state of Nevada, so I could likely just manually trace the outline of that state and call it a day - in fact that is my current plan next to fixing ERCOT.

It looks like pyiso won't give us the data for SPP, AESO, and IESO, which is quite a pity given that SPP is a huge market, AESO is relatively large, and IESO is both huge and rather notable for having completely phased out coal. Is it feasible to obtain the data for those sources via another method? I know IESO makes their data relatively easily accessible and I would personally really like to see them included. SPP/AESO are less easily accessible but it looks like the data could probably be obtained with some reasonable effort. Ideally I want to do the AESO/IESO (primarily IESO) borders as well, but I'll likely shelve that until it's possible to confirm that the borders would actually be used (so that there is data available for use with the shapefiles).

I have evaluated numerous other ISOs/RTOs in the US/CA and unfortunately they're pretty much all not going to be coverable. This excludes a substantial amount of power demand, generation, and interchange - from very large sources to relatively small ones. BPA, TVA, Duke, SOCO, FLA, PSCO, WACM, PACE, BANC, LDWP, PGE, PACW, PSEI, AVA, AECI, LGEE, SCEG, SC, I haven't looked into Canada in much more detail yet but I know Hydro Quebec is apparently not sharing their data, so that's a major blow.

I have a few possible ideas for ways to format the shapefiles that would reduce the visibility of the misalignment. I have already attempted to determine if the misalignment was due to mapping/resizing/projection mistakes (which took a while to run tests for), and so far it appears that none of these are at fault. I think the original data is flawed and it's just much more visible after scaling inwards. But anyways, worst-case we can just use the thick borders like I did, they're not too ugly at high zoom levels.

I'll download ne_50m_admin_1_states_provinces_lakes to get a better idea of how this will look in production.

So, optimistically, I should be able to deliver the following (italicized = highest priority):

  • CAISO (√)
  • ERCOT (revision needed to area/boundaries)
  • SPP (data source fix needed)
  • MISO (√)
  • PJM (√)
  • NYISO (√)
  • ISO-NE (√)
  • NVEnergy (area definition needed - this is the entire state of NV)
  • IESO (data source fix needed; area definition needed - this is the entirety of Ontario)
  • AESO (data source fix needed; area definition needed - I think this is the entirety of Alberta?)
  • All cross-border electric crossings (no data on ratings/throughput though - you might just discard this data anyways in favor of ISO/RTO figures and some guesswork or artistic placement of grid interties.)

This would be plenty for a good sample of major US utilities, assuming we can get alternative data sources for those three areas. Without alternative data sources I think we'll have a major issue when it comes to Canada due to lack of coverage outside of part of Manitoba. I have no clue how you'd handle between-ISO/RTO transmission, but I've got inter-border transmission marked and will include that shapefile when I'm finished, so there's something at least.

I can't give an ETA on completion of the remaining 2 primary tasks on my side (ignoring the 3 data source issues), but I'll try to get it done soon. I'm not sure how easy it'll be to integrate ne_50m_admin_1_states_provinces_lakes with 9-11 additional overlays (each as its own shapefile) for NA data if you're used to solely using ne_50m_admin_0_map_subunits for EU data but it should still be quite doable. It's not an issue in qGIS but I'm not sure how much that applies with the web app. Anyways, as soon as the two regions left are ready, I'll export each shapefile individually and send them all here.

@corradio
Copy link
Member Author

corradio commented Mar 1, 2017

Sounds good! Let's discuss on Slack the smaller details. Let's start small (maybe with only one ISO to begin with) and then we can extend slowly, instead of taking a mouthful too big :)

Olivier

@jacobbaratta
Copy link
Contributor

jacobbaratta commented Mar 3, 2017

The first attached zipfile contains shapefiles for ERCOT (based on EIA map), ISO-NE (based on EIA map), MISO (based on EIA map), NYISO (based on EIA map), PJM (based on EIA map), SWPP (based on EIA map), MISO_CA (the part of MISO that's in Manitoba - not based on EIA map (based on MISO's map)), and NVEnergy (I just manually traced the entirety of Nevada - it isn't 100% accurate but it's close enough that we can use it - not based on EIA map (not sure exactly what source I used to verify the boundaries but I'm confident that it's accurate enough)). The second one contains the CAISO shapefile and was obtained directly from a public source (it is a bit too detailed IMO due to the inclusion of other balancing regions but I didn't feel like mapping yet another region from the EIA map - I'll probably end up doing that at some point, but this is a good enough placeholder until then). Please evaluate them and see if they are of any use. My styling in QGIS will not carry over so that will need to be reimplemented to obscure the gaps between region borders. Note that all files in my zipfile use EPSG 3857 except for MISO_CA and NVEnergy, which use EPSG 4326. The CAISO shapefile (separate zipfile) uses EPSG 3310. For reference, ne_50m_admin_1_states_provinces_lakes uses EPSG 4326.

I'm still investigating the cause of the errors with my projections. It seems that there are quite a large number of contributing factors, and so far I have been unable to devise a working solution that increases accuracy to the necessary extent, although I have a few more ideas to explore before giving up. With 0.75mm solid line widths the errors in the projection mostly vanish for viewing at the scales it would likely be used at online, so that may be a solution for the interim.

I have double-checked all other major ISOs/regions - the only notable candidates with data available are BPA (unfortunately I cannot map their area accurately via any realistic means), AESO (area can be manually mapped relatively easily but data source isn't directly available via the library you use), and IESO (area can be manually mapped with some difficulty but data source isn't directly available via the library you use). Some data for other ISOs/regions could possibly be obtained via the EIA (as the organizations do not release it directly) but I'm assuming this approach is not at all viable and have scrapped all plans for mapping those regions. I'll try to look at pyiso at some point to see if adding AESO/IESO handlers is actually realistic, I believe it is, but I have not tested anything yet.

Anyways, here are the files. Hopefully they're usable.

all_self_SHP-v1.zip
Balancing_Authority_Areas.zip

@jacobbaratta
Copy link
Contributor

Conclusions regarding inaccuracy with my projections:

  • Inaccuracy comes from a lot of sources
  • Much of the inaccuracy is unavoidable. My base dataset is one of the EIA's SVGs, which had to be cleaned up, converted, opened in Illustrator, cleaned up again, then processed to extract every relevant (now wireframe) region of interest, each of which was multiplied by a fixed scaling factor, roughly centered, then exported as a maximum-resolution png file. This file was then imported into QGIS as a raster for manual georeferencing, then converted to a vector/shapefile. GIS is extremely complicated and not entirely intended or able to address this use case very effectively.
  • Issues with the accuracy of common projection types are likely responsible for a large portion of the observed error. The Pseudo Mercator projection in particular is notorious for causing issues (although literally every projection type has some sort of issue with it, they're all tradeoffs, and the pseudo mercator is the defacto standard). At least some of the observed errors are attributable directly to the distortions caused by this projection technique. Unfortunately I have no viable working alternative - I have successfully produced rasters using other coordinate systems/projection types, but I would have to remap literally everything to get those to work somewhat well, and distortion becomes an unexpectedly major issue when doing this (for different reasons). For all intents and purposes, it seems that the Pseudo Mercator is the best coordinate system/projection technique for my uses, flawed though it may be.
  • Very slight imprecisions in image scaling in Illustrator could have affected the result significantly.
  • The settings used while creating georeferenced rasters may have a significant influence on the accuracy of the result, but it is difficult and very time-consuming to assess the impact of different options here. So far I have had little success with modifying these settings (which I have tried pretty extensively).
  • Issues with the mismatch/difference in quality/simplification between the EIA files used and my reference maps used to georeference rasters, compounded by projection distortions, make aligning the rasters (and resulting vectors) very difficult.
  • Adjustment of raster/vector size/borders is a no-go so far. QGIS does not seem to handle this very well when automated (freezing for excessive periods of time when running affine transformations for example), and manually editing the sheer number of points per ISO/RTO outline is impractical (and also an issue due to the potential to introduce further inaccuracy in the process). I was hoping affine transformations would be a possible way to get the inaccuracy reduced a bit, but they operate far too slowly to utilize effectively (single-threaded only, extremely slow execution despite my fast single-core performance). It may be that the reason for the slowness of affine transformations is related to my specific platform, but I doubt this and suspect it's primarily related to the number of points and the algorithm used. Attempts at simplifying the number of points haven't gone well. You would think that an affine transformation would be absolutely perfect for GPU acceleration (via CUDA perhaps?), but QGIS does not seem to support this. At any rate, affine transformations are not the holy grail, they would be a partial fix at best, so I suppose it doesn't matter too much.
  • Manual realignment of the entire raster/vector is a no-go due to distortion issues. Affine transformations or painstaking point-by-point adjustments are the only viable ways to improve the amount of error observed, and as explained above, neither are particularly viable options at the moment.

So, here's the overall conclusion:

  • For North America, a different approach will need to be taken than was taken for the EU. Unlike the EU, you can't just neatly slice things up along country lines - you need overlays in virtually every case, and those are not going to align very prettily.
  • As the inaccuracy currently has no viable solution, it will have to be dealt with. This is doable.
  • The complexity of the script needed to generate the (GeoJSON?) files is unavoidably going to increase for the US map, and will likely require stylistic tweaks (not sure where these would be applied) to obscure the inaccuracy with the vectors.
  • If vector inaccuracy is obscured to a significant enough degree (depends on zoom level), this vector data is quite definitely an acceptable approximation of a number of the most major US/CA ISO/RTO boundaries that can be used for a US/CA version of electricitymap.

Tl;Dr: The inaccuracy in the data is extremely complex in origin and very much unavoidable right now, but it can still definitely be obscured, and if it is obscured, this data should easily be sufficient for an initial US/CA version of electricitymap. I am out of ideas for resolving the inaccuracy problems so I'm not planning on spending much more time on that in the immediate future - what files I uploaded as the best we've got unless someone's been hiding a bunch of really high quality shapefiles somewhere hidden from Google.

@corradio
Copy link
Member Author

corradio commented Mar 4, 2017

Hi @jacobbaratta,

Thanks for the tremendous effort. I'm currently working on making a mobile version of the map, but once I get a bit more time I'll try to see if I can integrate your shapefiles. If anyone wants to give a go at integrating the shapefiles into the map, please reach out on Slack and I can help get you started.

Hopefully this will work!

@corradio corradio changed the title Add US data Split US May 2, 2017
@jacobbaratta
Copy link
Contributor

@alixunderplatz I have a bunch of resources like that NERC map that I can share if people are interested, I collected quite a lot of different resources to use as references while working on my original shapefiles.

@HansHyde I saw those linked on eia's site a while back and went through them. Some of them are quite nifty, and it's the first time some of that data has been officially released in shapefile format, but there's nothing new in there that is of actual value to us for this particular project. Their ISO-level shapefiles are no more accurate than the ones I created/collected (if anything they are considerably cruder and less accurate), which is a major disappointment. Their only benefits are that they have nice metadata (not really relevant for our use case) and that they may seem somewhat more "official" (which is a subjective argument, not an objective one).

@corradio, has anyone made any progress with merging third-party shapefiles/geometries into the map? I've given it a try once or twice a while back and made no significant headway before giving up -- I'm just not familiar enough with the methods and libraries you're using, and there's no obvious way to handle merging additional shapefiles/geometries (plus it's not just getting it to display, but also getting it to work as usable regions, since much of the app appeared to rely on hooking directly into attributes in the original geometry to tell regions apart and know what data goes to what area, etc).

@systemcatch
Copy link
Collaborator

@jacobbaratta I was able to split the Canary Islands using new shape files in #1022. @corradio & @maxbellec then really improved how the map generates topologies in #1064.

You might as well add some of your resources to this issue thread so people don't end up repeating work.

@systemcatch
Copy link
Collaborator

Some really good news everyone!

I found a shapefile showing Balancing Authority areas for the whole of the USA, https://hifld-geoplatform.opendata.arcgis.com/datasets/0242ddfc59554fadb8ffe8066c573c93_0

The attribute data for the shapes is very detailed. After a lot of work with qgis, topogen.sh and generate-geometries.js I've been able to make this.

usa_by_ba

@maxbellec
Copy link
Contributor

Not the most beautiful map, but worth gold 😍
Send the PR!

@alixunderplatz
Copy link
Collaborator

@systemcatch Very nice! Seems like all of the areas more or less overlap with (many unused) others, which causes that "disintegration", esp. for CAISO and SPP?

Have you looked into the other shapefiles on that site already?

I really like the one for the large ISO areas. Maybe that could reduce the fragmentation (and required effort) to some extent?
https://hifld-geoplatform.opendata.arcgis.com/datasets/9d1099b016e5482c900d657f06f3ac80_0

image

@systemcatch
Copy link
Collaborator

systemcatch commented May 14, 2018

@alixunderplatz I had a look at some of the other shapefiles but they didn't seem to represent the data parsers are currently getting very well. The large ISO areas are good but big parts of the USA are missing and wouldn't show up on the map. We'd have to layer the ISO's onto a base shape of the US (:thinking: possible?) and find shapes for BPA and IPC.

As regards the image I posted there's problems with overlapping and highlighting everywhere. A big question is whether some of the balancing authorities contained within other BA's are included in the larger BA's data from the parsers. If this is the case then the smaller areas can be dropped.

@jarek
Copy link
Collaborator

jarek commented May 14, 2018

I think we can be selective, so including for instance NY / New England from Natural Earth and SPP from a shapefile. Just as long as they line up decently. In the screenshot above a part of Maine on the border with New Brunswick got flooded ;) 🌊

@brunolajoie
Copy link
Contributor

brunolajoie commented Dec 12, 2018

I've found a crazy map of the US grid. http://www.eei.org/about/members/uselectriccompanies/Documents/EEIMemCoTerrMap.pdf

Does anyone understand the difference between all these "EEI companies", Utilities, ISO grids, IPCC? I'm starting to get lost with the US power system ;)

image

@systemcatch
Copy link
Collaborator

My understanding of that map is that those are all for profit companies that serve consumers. I recognize some of the companies in California that are part of CAISO, doesn't look to be any electric coops on that map.

@corradio
Copy link
Member Author

@systemcatch should we close this issue?

@robertahunt
Copy link
Contributor

Hey @systemcatch @corradio , I found some (new? I didn't see them in the above thread) geo data for the US. This one defines the different control areas, instead of independant system operators. Is this new? Is something we can use to fill out the map? and if so, what are some next steps? https://hifld-geoplatform.opendata.arcgis.com/datasets/control-areas/data?page=1

@robertahunt
Copy link
Contributor

robertahunt commented Jan 28, 2020

At least most of the control area names seem to correspond 1-1 to the names of the balancing authorities which we can now get hourly production data per fuel from eia here: (https://www.eia.gov/opendata/qb.php?category=3390101)

Update:
https://hifld-geoplatform.opendata.arcgis.com/datasets/electric-planning-areas/data also seems to contain similar areas, and has abbreviations that correspond directly to the BAs, but contains quite a few areas which eia does not have data for (example, Alabama power company)

@systemcatch
Copy link
Collaborator

Hey @robertahunt I remember these areas from when I tried to build the USA from them (#143 (comment)). The main difficulty was dealing with all the overlaps and enclaves.

One possibility is to take the shapes we want (MISO, PJM etc etc) and overlay them onto the whole USA with statelines vaguely showing as a guide. Something like this can be done I think (https://medium.com/@mbostock/command-line-cartography-part-1-897aa8f8ca2c)

@robertahunt
Copy link
Contributor

Nice!! So I have only gotten so far as getting all the geojsons to display on the map (like you had before). Now, we are considering how best to display this information on the map.

I just wanted to document some of the research we are doing now:
Due to the overlaps, we are trying to ensure that each balancing authority is actually it's own electric grid - so that our assumption that they act like a copper plate within that grid is valid. My opinion on this after doing some research is that yes, each BA has it's own associated grid, and even though these grids overlap geographically, they are using different transmission lines in reality - With the exception that there is something in the US regulations called 'Overlap Regulation Service', which seems to be when one BA contracts another to take care of the physical transmission of energy in an area, however I have not been able to find any examples of BAs that do this, or much information on this at all.

Documentation:
NERC standard for Real Power Balancing Control Performance (indicates each BA is responsible for grid frequency within it's control area:
"Each Balancing Authority shall operate such that, on a rolling 12-month basis, the average of the clock-minute averages of the Balancing Authority’s Area Control Error (ACE) divided by 10B (B is the clock-minute average of the Balancing Authority Area’s Frequency Bias) times the corresponding clock-minute averages of the Interconnection’s Frequency Error is less than a specific limit. This limit ε12 is a constant derived from a targeted frequency bound (separately calculated for each Interconnection) that is reviewed and set as necessary by the NERC Operating Committee."
https://www.nerc.com/pa/Stand/Version%200%20Relaibility%20StandardsRD/Draft_4_VO_Clean_Operating_Standards_01_07_05.pdf

Overview from NERC on balancing and frequency control.
"Each Balancing Authority in an Interconnection is connected via high voltage transmission lines (called tie-lines) to neighboring Balancing Authorities. "
https://www.nerc.com/docs/oc/rs/NERC%20Balancing%20and%20Frequency%20Control%20040520111.pdf

Document on interface between MISO-SPP, lists 171 high voltage interfaces between the two:
https://www.spp.org/documents/59006/spp-miso_rsc_oms_response_spp_miso_final_v3.pdf

@robertahunt
Copy link
Contributor

After much fighting with the US geojsons, I have a couple options for how to remove the overlaps from the balancing authorities (for display purposes only)

Option 1: Always show the smaller balancing authority
ezgif-2-e3d0366a7205

Option 2: Try to find smart ways to assign approximately half of the overlapping area to one BA and the other half to the other BA.
ezgif-2-a685aa5045b4

Thoughts? Are either of these good enough for visualization purposes?
I think we still need the overlapping geojsons for latitude and longitude queries, and bounding boxes

@robertahunt
Copy link
Contributor

I think we can close this now?

VIKTORVAV99 pushed a commit to VIKTORVAV99/electricitymap-contrib that referenced this issue Jan 13, 2023
madsnedergaard added a commit that referenced this issue Jan 30, 2023
* Adds .vscode files and a symlink

* Adds temporary dev-easement-rule-disablements

* Makes magic-numbers rule more chill

* fixes magic number

* Removes unused disable-comments

* Moves testing utils and setup to separate folder

* Groups utils in one folder

* Restructures app

* Fixes imports

* capacitor scripts

* Moves files around

* Fixes paths to get tests passing

* Fixes linting paths after renaming tests

* Updates VSCode settings to include file nesting

* block all crawlers (#11)

* block all crawlers

* add no index meta

* initial map setup (#13)

* remove cz (#12)

* add tailwind classname package (#10)

* update pnpm lock

* Adds error overlay to display runtime errors (#16)

* Moves integration tests over and makes them work again (#15)

Co-authored-by: Tony <tonyvanswet@gmail.com>

* Adds basic layout (#17)

* Adds deploy script (#14)

* Tonyvanswet/ele 1338 UI header component (#20)

* Adds some mockdata with new structure (#22)

* add mockdata

* fix

* LeftPanel UI (#18)

* Replaces tailwindcss-classnames with clsx.
See https://linear.app/electricitymaps/issue/ELE-1331#comment-5816d8f5

* Ensures tailwind autocomplete stil lworks

* Adds missing dependency

* Updates outer wrapper styles

* Adds initial work on the left panel container

* Fixes shadows

* Adds fixed to main

* Uses proper HTML tags

* Removes clsx usage

* Adds state with atom and initial timecontroller UI (#21)

* Adds libraries

* atom logic

* change line width to 80

* Update api calls

* adds initial ui for timecontroller

* Jotai usage documentation

* Readme

* incorporate feedback

* Update .prettierrc.json

* lint

* Update README.md

Co-authored-by: Tony <tonyvanswet@gmail.com>

* Tonyvanswet/ele 1371 map UI basic geometries and colors (#19)

* initial map setup

* get map rendering

* crop antartica with very magical numbers

* gray map and many eslint errors

* add mockdata

* fix

* map with types

* use v6 api

* basic map

* Update web/src/hooks/theme.ts

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Update web/src/api/getState.ts

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Update web/src/features/map/map-utils/generateTopos.ts

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

Co-authored-by: Markus Killendahl <markus.killendahl@tmrow.com>
Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* add all the flags (#24)

* Zone Details: Header part 1 (#23)

* add all the flags

* Add zoneFlag with hardcoded return

* Change leftpanel styling

* Add initial zoneHeader

* Works for countries

* Disable eslint

* Disables the annoying import-organize

* Lints everything with 90 char line-width (#29)

* Updates prettierignore

* Removes unused overwrite

* Ensures script uses prettierignore lisr

* Disable the horrible "auto-removing imports"

* Ignores config json files

* Formats everything with printwidth 90

* Updated map logic (#27)

* WIP

* WIP 2

* setup types

* Fix test

* Remove unused things

* Disables the annoying import-organize

* Split theme types

* Color on all effects

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Zone Details Header: Part 2 (#25)

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Sets up basics for dark-mode (#28)

* Adds translations (#31)

* add locales

* add translation logic

* use translation hook in timecontroller

* Adds v6 to mockserver (#30)

* Updates script to create paths and use v6

* Removes mock data for v4 and v5

* Adds mock data for v6

* Updates mockserver to work with v6

* Style toggle group

* inject dependencies in toggle

* style and refactor timeslider

* Change header to be fixed positioned

* Remove error and loading handling on map. It should ideally always be drawn.

* Close left panel

* Add timeaxis

* refactor timecontroller

* add helpers

* remove magic number rule

* add clock icon in html file

* install react spinners

* change tailwind config. Should confirm best practice.

* add slider thumb custom icon

* Basic map interactivity: Hover (#33)

* Ensure atom returns typed value

* Timeslider (#35)

* remove unused config

* change tailwind config

* use brand green

* remove ternary operator

* incorporate feedback

* Fixed layout problems with Header (#36)

* Makes header not-fixed and adjusts link font size

* Makes time controller fixed (instead of absolute)

* Basic map interactivity II: Click & Selected (#34)

* Ranking panel (#32)

* add locales

* add translation logic

* use translation hook in timecontroller

* Copied files from #26

* working prototype

* ensure timecontroller is on top

* ranking rows

* refined ranking panel

* remmove undefined

* Adds onClick handler

* Applies curly rule

* Update web/src/features/panels/ranking-panel/ZoneList.tsx

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Creates and uses an InternalLink component that preserves state

* remove ref and update hover

Co-authored-by: tonypls <tonyvanswet@gmail.com>
Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* add arrows (#41)

* Set up icons and replaces existing usage (#37)

* Documents icon usage

* Adds react-icon and documents usage

* Replaces icons with new approach

* Tonyvanswet/ele 1375 UI set up prodcon and aggregate toggle (#38)

* start toggle

* reusable toggle

* toggle with translations

* update enums

* Exchanges: map rendering (#40)

* add arrows

* add exchange types

* added key

* add scales helper

* add custom layer

* add exchange logic

* add to map

* Update web/src/features/exchanges/ExchangeArrow.tsx

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* Update web/src/features/map/Map.tsx

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* Update web/src/features/map/Map.tsx

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* Fixes locales path

* start on buttons and tooltips (#39)

* start toggle

* reusable toggle

* toggle with translations

* update enums

* start on buttons and tooltips

* darkmode + zoom controls

* add zoom toggles

* zoom darkmode

* map controls ready for review

* Remove theme check

* Update web/src/features/map/Map.tsx

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Adds translation scripts (#43)

* Tonyvanswet/ele 1378 UI basic tooltip component (#44)

* start toggle

* reusable toggle

* toggle with translations

* update enums

* start on buttons and tooltips

* darkmode + zoom controls

* add zoom toggles

* zoom darkmode

* map controls ready for review

* tooltip follows cursor in country

* empty tooltip

* tidy

* A bunch of minor improvements (#46)

* Adds LoadingOverlay (#47)

* Add initial area charts (#48)

* install d3

* modify types

* tweak timeaxis

* set up helpers

* add chart components

* install currency lib

* add two additional graphs

* Charts v2: Refactor with hooks (#49)

* install d3

* modify types

* tweak timeaxis

* set up helpers

* add chart components

* install currency lib

* add two additional graphs

* change usegetzone hook to no params

* refactor pricechart

* refactor carbonchart

* refactor breakdown chart

* extend types

* change areagraph data format

* set it up

* feedback

* Allows ts-ignore comments with descriptions

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Adds new Flag component (#50)

* some theme colors and improve darkmode (#52)

* no user select for buttons / controllers (#56)

* less fruits, more clickable map (#55)

* Tonyvanswet/zone tooltip data (#51)

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Changes title component to incapsulate logic and use real data (#54)

* Adds new Flag component

* Updates flag props

* Changes title component to incapsulate logic

* Changes gauges to use 0-1 scale instead of 1-100.
As our data comes in this way and the charts needs it like this anyway.

* Changes CountryTag to only show on subzone

* Adds new TimeDisplay that uses atom directly

* Adds comment (and waits with implementation)

* Hooks ZoneHeader up with more data

* Sets default selected datetime as latest

* Bumps version

* Removes flags

* Charts v3: Emission mode (#53)

* install d3

* modify types

* tweak timeaxis

* set up helpers

* add chart components

* install currency lib

* add two additional graphs

* change usegetzone hook to no params

* refactor pricechart

* refactor carbonchart

* refactor breakdown chart

* extend types

* change areagraph data format

* set it up

* feedback

* Adds emission chart

* Set up display by emissions

* fix typos

* incorporate feedback

* fixes Markus' fix

* Fixes merge conflict mistake

* Fixes merge conflict issue

* bump

* tidy up and some branch feedback (#58)

* Tonyvanswet/ele 1479 make language selector work (#57)

* start on language selector

* language select

* Refactor charts further and adds storybook (#59)

* Adds basic setup

* Adds default sample stories

* Adds example MDX

* install d3

* modify types

* tweak timeaxis

* set up helpers

* add chart components

* install currency lib

* add two additional graphs

* change usegetzone hook to no params

* refactor pricechart

* refactor carbonchart

* refactor breakdown chart

* extend types

* change areagraph data format

* set it up

* feedback

* Adds emission chart

* Set up display by emissions

* fix typos

* WIP

* WIP 2

* WIP

* Fix merge conflicts probably

* Remove timeaverage as param

* Add on hover effect to breakdown chart

* Support storage

* Add storage story

* Fix import

* reset datetimes correctly

* Incorporate feedback

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* onboarding modal (#60)

* onboarding modal

* remove dialog package

* use local storage atom

* Update web/src/components/modals/OnboardingModal.tsx

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* Update web/src/components/modals/OnboardingModal.tsx

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* Update web/src/components/modals/OnboardingModal.tsx

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* Update web/src/components/modals/OnboardingModal.tsx

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* updates from merge

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* Tonyvanswet/fix easy todos (#62)

* some TODO tidying

* get exchanges working again

* Update Map.tsx

* Tonyvanswet/fix todos part2 (#63)

* some TODO tidying

* get exchanges working again

* small fixes

* update theme type

* Get e2e tests working (#64)

* Ensure maptooltip does not remount and call api

* stash

* test passes

* Restructure mockserver

* remove stories

* remove

* Bar Breakdown Chart (#65)

Co-authored-by: Markus Killendahl <markus.killendahl@tmrow.com>

* Fixes tooltip issue + broken bar breakdown chart (#69)

* Fixes tooltip issue

* Reverts change in data from hook

* FAQ Panel (#68)

* Tonyvanswet/ele 1370 set up new version logic (#61)

* start on toast

* new version toast

* new version with longer toast and descriptive refetch intervals

* check if prod before displaying new version toast

* prettier

* Tonyvanswet/update ranking panel style (#71)

* Adds outer panel

* Adds FAQ content

* Adds Tailwind plugins and animations

* Adds Poppins font

* Updates panel styling to handle scrolling and fixed containers

* Accepts FAQ in filename

* update styles based on feedback

* update after merge

* update scroll bar style and finish ranknig panel

* Adds dark mode hover styling

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Dark-mode of bar-breakdown + chart titles + minor adjustments (#70)

Co-authored-by: Tony <tonyvanswet@gmail.com>

* Updates to handle new v6 data structure (#72)

* Updates to handle new v6 data structure

* Updates mock data

* Adds correct steps in CO2 scale (#73)

I assume the stuff was done to avoid magic numbers, but we removed that rule

* Small screen tweaks (#75)

* Tonyvanswet/ele 1508 carbon intensity legend (#74)

* start legend

* legendary

* move styling to legend container

* move styling to legend container actually saved this time

* rename legend container

* Adds areagraph tooltips (#67)

* Add shared tooltip component

* store zonedetails in meta

* share function from maptooltip

* add simple tooltips to charts

* Update locales

* Remove boiler story

* add keys to zonedetails

* add helper components

* add helper methods

* add breakdown tooltip

* add stories

* add other tooltips

* fix some bugs

* set up emission chart tooltip

* handle different size of tooltip placement

* log statement

* make tooltip more responsive

* style fixes

* Remove duplicate question mark

* add a debug file

* add a unit test

* clean up

* Sets Vite up to build index.html using a template (#77)

* Finalise Ranking Panel (#78)

* Tonyvanswet/ele 1368 set up solar and wind layer (#80)

* borked wind layer

* sort of works

* add todo on moment

* removed moment, kinda working

* remove moment from packages

* Removes cleanup function which is not allowed on useCallback

* Fixes hour offset (no clue why though...)

* Removes unused import

* Deletes unused hooks

* Removes react-transition-group

* Removes another cleanup-function

* Replaces fade-in transition with tailwind

* Updates data-fetching logic:
- use correct dates
- type data response
- retry with prev. timestamp if it 404s

* Uses data type

* Uses singleton pattern for Windy instance + types

* Update web/src/api/getWeatherData.ts

* Removes commented-out code

* Moves scales to different files

Co-authored-by: Markus Killendahl <markus.killendahl@tmrow.com>
Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Set up mobile tooltips (#83)

* update spring bottom sheet

* add mobile styling

* remove comment

* use tailwind numbers

* Add zone detail states (#79)

* Add empty bar breakdown chart

* add more states and refactor

* add contributor list and source

* ensure graphs are not rendered when no data

* remove boilerplate story

* feedback

* Merge conflict and remove warning

* Refactors Map to avoid excessive rendering (#81)

* Changes tooltip to read from global state

* Creates new MapWrapper to split components out

* Moves map-related atoms inside feature-folder

* Moves shared map types to its own file

* Moves state to global atoms

* Refactor the onMouseMove handler for readability

* Reads isMoving from atom

* Makes panel open when clicking a country

* Reemoves types that can now be inferred + minor tweaks

* Hides tooltip when dragging map (like before)

* handle conflicts

Co-authored-by: Markus Killendahl <markus.killendahl@tmrow.com>

* Fix SE-SE4 not being hoverable (#84)

* add empty zone to geometries

* add comment

* use zoneid as featureid

* format

Co-authored-by: Markus Killendahl <markus.killendahl@tmrow.com>

* starting to work through errors and warnings (#86)

* Add geometry and zone-config scripts (#85)

* WIP

* executable through ts-node

* Update script

* reintroduce eslint rules

* add eslint rules back

* use current config folder

* use current config

* Add tooltip to barbreakdown chart and style zonedetails a bit (#87)

* Add tooltip functionality

* style zonedetails

* rename variable

* Adds solar layer 🌞 (#90)

Co-authored-by: tonypls <tonyvanswet@gmail.com>

* Adds loading indicators on weather buttons (#89)

* fixes atom name usage

* Changes layer orders

* Removes vite-plugin-html and uses a plain index HTML (#94)

* Removes variable usage from index file.
Also stops Prettier from breaking long lines

* Removes package

* Switches to variable font for Inter

* BreakdownChartTooltip bugfixes (#93)

* Disable test coverage

* test passes

* returns 0 instead of ? for existing zero production

* fix darkmode and positioning

* refactor tooltip calculations

* tidy left panel (#88)

* Tonyvanswet/ele 1366 set up proper sentry process (#91)

* add sentry

* further sentry updates

* sentry routes

* add sentry auth

* Update index.html

* Update main.tsx

* update project name

* Adds colorblind-mode (#92)

* excludes subzones (#95)

* fix style (#96)

* extra space (#97)

* add tooltips (#99)

* add dark mode bg for overlay (#100)

* 🎁 Update bottom sheet snap points (#102)

* reduce snapsshots

* align text and datedisplay

* solar and wind legends (#103)

* solar and wind legends

* hide legends on small screens

* tidy legends

* 🎁 Display estimated and aggregated badges (#98)

* display badges

* add height

* 🎁 Improve tooltip interactivity on mobile (#101)

* add x button on areagraphtooltip

* remove hover layer effect on mobile

* add a bit more styling

* works on larger devices

* change top padding to come from styles and add z index to go above timecontroller

* ensure overlay above timecontroller

* add none pointer events to prevent flickering

* close tooltip when zoneclicked

Co-authored-by: tonypls <tonyvanswet@gmail.com>

* Adds custom events (#108)

* Mobile modals part I: The base (#104)

* Mobile modals part II: Settings (#105)

* Mobile modals part III: Info (#106)

* use latest inter (#109)

* Updates Sentry project name + check for env vars

* Tonyvanswet/add alaska translation (#110)

* add zone name for Alaska

* add alaska and prevent long zone overflow

* onboarding modal for mobile (#113)

* onboarding modal for mobile

* Adds padding below titles and switches font

* Adds shadow to modal buttons

* Makes modal buttons overlap modal on mobile.
This gives more space for content

* Positions modal closer to middle of screen on desktop

* Makes modal less high on mobile + increases shadow

* Constraints logo size on first slide

* Ensures TimeController is always rendered.
Otherwise the loading spinner shows again.

* Adds comment to timecontroller

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Fix broken refresh on app (#115)

* Fixes tooltip crash (#118)

* Fixes issue when adblocker is used + adds types (#117)

* Disables ts-checking on Windy and .stories.*

* Rearranges and updates images and icons (#114)

* fly to places (#116)

* fly to places

* Update web/src/features/map/Map.tsx

* update from feedback

* update initial view

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* disclaimer (#111)

* disclaimer

* update disclaimer

* change title width for difference screen sizes

* Adds a tiny bit of padding on mobile

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Updates scripts (#112)

* Mobile modals part IV: FAQ (#107)

* Changes atoms to only use localStorage (#119)

* Tonyvanswet/ele 1538 enable center on callerlocation (#120)

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* allow toggle between production and consumption (#121)

* Fixes skip-onboarding

* Disables PWA and unregister SW temporarily

* touchable tooltips for toggle component (#122)

* Renames web

* Renames web-old to web while merging

* Reverts web folder renaming

* Merges locales

* Updates generated config files

* Adds scripts to ensure we generate new zone config

* co2 scale ++ (#126)

* co2 scale ++

* testing gray map

* remove test code, prevent rotate and tilt

* Update web/src/hooks/colors.ts

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* capitalize tooltip (#127)

* add get our data link (#123)

* add get our data link

* update from feedback, remove message css

* Update web/src/features/charts/ChartTitle.tsx

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>
Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>

* unhover zone when clicked

* fix jobs link (#132)

* filter breakdown chart exchanges for selected aggregate (#133)

* feedback from Rasmus (#124)

* correct emmsions tooltip (#130)

* hardcode center location for FJ US-AK and RU-FE, and Offset center (#136)

* Updates error boundary message (#128)

* time controller german translation (#131)

* Generates index files for all paths! (#135)

* Tonyvanswet/ele 1572 fly on zone click either get it to move (#137)

* hardcode center location for FJ US-AK and RU-FE, and Offset center

* update world json

* Ensures index-generating script runs on deploy

* Fixes bug

* Bumps version

* Fixes minor design feedback

* Use manual chunking to split large chunks (#139)

* show production / consumption in carbon tooltip (#134)

* show production / consumption in carbon tooltip

* Update web/src/features/charts/tooltips/CarbonChartTooltip.tsx

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* more readable conditional

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* check for zonedata before getting exchanges (#142)

* Tonyvanswet/ele 1581 exchanges are still shown when in (#143)

* Tonyvanswet/ele 1579 better message for no data tooltip (#141)

* add spacing for units (#140)

Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Styling tweaks + Deselect zone when going back to map on mobile (#138)

Co-authored-by: Tony <tonyvanswet@gmail.com>

* Adds tooltip to LowCarbon Gauge in panel (#144)

Co-authored-by: tonypls <tonyvanswet@gmail.com>

* Improves no-data tooltip (#147)

Co-authored-by: tonypls <tonyvanswet@gmail.com>

* Adds overlay to breakdown chart (#148)

* Round map to 4 decimals (#151)

* Use @turf/helpers (#150)

* Tvs rj/ux peerprogramming (#152)

* UX feedback merged with master

* use same font size for gauges in zoneheader and maptooltip

* merge classnames

* feedback + max width

* make things no selectable on mobile + enable touch zoom but disable r… (#153)

* make things no selectable on mobile + enable touch zoom but disable rotation

* tap hightlight comment

* Deployment improvements (#149)

* unhover other zones on click (#154)

* Adds comment about missing info in type

* Fixes incorrect script

* version bump

* fix fully renewable header (#159)

* Minor fixes (#158)

* Allows robots in

* Bumps version

* Adds missing translation

* version bump

* Tonyvanswet/ele 1621 deploy mobile apps (#156)

* make things no selectable on mobile + enable touch zoom but disable rotation

* tap hightlight comment

* unhover other zones on click

* app icons + android / ios specific changes

* disable pitch

* safe area time controlloer

* safe area left panel

* bump versions

* Moves all Storybook stories next to their components (#145)

* Unifies CO2 Intensity display (#146)

* Squares and gauges now uses translations (#161)

* Fixes /zone/WHATEVZ breaking the app + disables chunking for now (#160)

* TimeController improvements (#162)

Co-authored-by: Tony <tonyvanswet@gmail.com>

* Fixes multiple bugs (#164)

* Jl/license (#155)

Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>
Co-authored-by: Mads Nedergaard <nedergaardmads@gmail.com>

* Change div to picture to fix webp exchange arrows (#169)

* Tvs/i os build (#170)

* Disables transition of numbers (#171)

* Tvs/fixcharts ios 14 (#172)

* Fixes New Version Available + Reduces Sentry transaction amount (#173)

Co-authored-by: Tony <tonyvanswet@gmail.com>

* Removes console log

* More minor tweaks (#165)

* Updates configs after master merge

* Fixes search params

* Temporarily disables IN-NO

* Sets correct theme color

* Enables CI again (#163)

* Bumps version

* Tvs/sensible bounds (#174)

* adjust view bounds

* disable pitch and rotate even when no data

* Tvs/release candidate (#176)

* adjust view bounds

* disable pitch and rotate even when no data

* prepare for mobile app release

* prettify

* bump version

* stop replace all errors (#177)

* Remove unneeded fields from config files (#167)

* Replaces map with a fallback shown for users on old ios versions (#175)

* comment out atom dev tools (#178)

* Fix invalid href lang + add preconnects (#180)

* include new tracking events (#179)

* delete web-old

* Fixes link

* Removes new workflows for new

* Updates ignores

* Removes clsx

* Removes license from boilerplate

* Ignores vscode

* Removes vscode symlink

* Deletes deploy script

* Removes boilerplate leftover

* bring back site links to mobile app

* prettify

---------

Co-authored-by: tonypls <tonyvanswet@gmail.com>
Co-authored-by: Markus Killendahl <markus.killendahl@tmrow.com>
Co-authored-by: Viktor Andersson <30777521+VIKTORVAV99@users.noreply.github.com>
Co-authored-by: lav-julien <112700080+lav-julien@users.noreply.github.com>
@ktehranchi
Copy link

Hello- Are the cleaned-up North American Balancing Authority shapes available for download for academic research purposes? Thanks!

@VIKTORVAV99
Copy link
Member

Hello- Are the cleaned-up North American Balancing Authority shapes available for download for academic research purposes? Thanks!

We don't have any separate files for these but the geojson code for each zone is contained in web/geo/world.geojson.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests