An implementation of my MUGicalPHP calendar generation script that runs as a collection of AWS Lambda serverless functions.
If you'd like to subscribe in your calendar application of choice, there are two options:
These calendars include a separate event for every meetup. Depending on the city, that could involve a lot of events.
- https://krishoward.org/sydneymugs.ics
- https://krishoward.org/melbournemugs.ics
- https://krishoward.org/brisbanemugs.ics
- https://krishoward.org/goldcoastmugs.ics
- https://krishoward.org/perthmugs.ics
- https://krishoward.org/hobartmugs.ics
- https://krishoward.org/newcastlemugs.ics
These calendars have a single all-day event for each day that includes all of the meetups as a list within the description. For full details on a meetup, you'll have to click on the provided URL.
- https://krishoward.org/sydneymugs-digest.ics
- https://krishoward.org/melbournemugs-digest.ics
- https://krishoward.org/brisbanemugs-digest.ics
- https://krishoward.org/goldcoastmugs-digest.ics
- https://krishoward.org/perthmugs-digest.ics
- https://krishoward.org/hobartmugs-digest.ics
- https://krishoward.org/newcastlemugs-digest.ics
The basis for this app came from Open Austin's meetup-proxy-aws-lambda. I've broken it up into four separate lambda functions, each of which lives in its own directory within src
:
src/getcities
src/getgroups
src/getevents
src/makecalendar
There's also a lambda layer that includes common dependencies:
npm install
npm install aws-sdk
npm install request
npm install request-promise
npm install ical-toolkit
npm install striptags
There's also a special function that's used for refreshing your Meetup OAuth credentials when needed. It requires a couple other dependencies:
npm install express
npm install axios
As Meetup have deprecated use of simple API keys, we have to jump through a few hoops to get authorised. (I'm indebted to Soham Kamani's blog post for showing how to do this.) Start by going to Meetup and creating a new OAuth consumer. Set the Redirect URI to http://localhost:8080/oauth/redirect
. Once you've created the consumer, you'll have a Key and a Secret.
Go to auth/index.html
and replace {key}
with the Key for your Meetup consumer app.
Then go to auth/index.js
and replace {key}
with the Key and {secret}
with the Secret for your Meetup consumer app.
Open a terminal and go to the auth
directory and run node index.js
. Then open a browser window and visit http://localhost:8080, where you should see the index.html
landing page. Click on the “Login with github” link, and youll be redirected to the familiar OAuth page to register with Github. Go ahead and authenticate. Afterwards it'll try to redirect you to http://localhost:8080/oauth/redirect and just hang since that doesn't exist. However, have a look back in your terminal which should have logged four lines.
requestToken: {longcrypticstring}
accessToken: {longcrypticstring}
expiry: 3600
refreshToken: {longcrypticstring}
Copy and paste them all somewhere, and then you can Control-C to kill the webserver.
In the AWS console, go to Systems Manager and click on Parameter Store in the menu on the left. You're going to create four new parameters: accessToken
, refreshToken
, clientID
, clientSecret
. For each one, set Tier as standard and Type as Secure String. Paste the relevant values into the boxes for Value. (clientID
is your Meetup consumer Key, and clientSecret
is your Meetup consumer Secret.) You can leave all other fields the same.
If you want to test out one of the lambdas, you'll need to first install the AWS CLI to your machine.
Then update the tests/index.js
file to point to the lambda you want to test.
Then just run:
npm run test
First create the layer for your dependencies. Run npm run build
which will reinstall any missing dependencies and create a zip file called mugicalnode-layer.zip
. In the AWS console go to Lambda and then click the Layers option. Create a new layer MugicalNodeDependencies
with the runtime Node.js 10.x
. Click the option to upload the zip you just created and save the layer.
Next create the lambda functions. You can just use the "Author from scratch" option for each one and paste in the relevant function. The runtime should be Node.js 10.x
. The name of each lambda needs to match the name of a folder within src
. I recommend setting the timeout to 30 seconds for each. (Note: for makecalendar
you will likely want to bump the memory up to 256MB.)
I recommend setting up a new execution role. You'll need to make sure this role has access to CloudWatch Logs, S3, SQS, Systems Manager parameters (getParameter, get Parameters, and putParameter), and the relevant DynamoDB tables. (See below.)
Make sure your function is set up to use the layer you just created.
Note: if you later update a function locally, you can deploy to AWS by running this command (being sure to substitute in the name of your lambda/folder:
./lambda.sh {nameoflambda}
This lambda gets a list of cities from a database table. It's used by a Step Function that we'll define later...
- You need to create a DynamoDB table called
appconfig
with the primary partition nameappname
(String) and primary sort keymy_key
(String). - You will then create an item in the table that consists of a list of cities. The JSON is defined in
ddb-cities.json
. Just create an item and select the option for "Text" and "DynamoDB JSON." Then you can paste in the JSON from the file.
This lambda gets all the meetup groups associated with various topics in a given location and sends them to an SQS queue. It's triggered as part of the Step Function we'll define.
- You need to create an SQS queue called
meetupgroups
.- The retention period on the queue is set to 23 hours, which ensures that if something goes wrong, it gets purged before it runs again the next day.
- The default visibility timeout on the queue is set to 1 minute.
- In the Redrive policy, set the Maximum Receives to 20 and set a dead letter queue for any groups that fail.
- Update the
QUEUE_URL
in the lambda function to match the URL of your queue. - Make sure the region in the lambda matches the one where you set up your SQS queue and your parameter store!
- Make sure your lambda execution role has permissions to access the parameter store!
- You also need to add the list of topics as an item to your
appconfig
DynamoDB table. The JSON is defined inddb-topics.json
. Just create an item and select the option for "Text" and "DynamoDB JSON." Then you can paste in the JSON from the file.
This lambda retrieves a group from the queue and then calls the Meetup API to retrieve its upcoming events. These are saved to a DynamoDB database. This lambda is triggered from the meetupgroups
SQS queue. Events will be deleted from the database a day after they finish.
TODO: if authorisation fails, refresh accessToken with Meetup...
- You need to create a DynamoDB table called
meetups
.- You should enable the
Time to live attribute
and set the TTL attribute todeletedate
.
- You should enable the
- Make sure the regions in the lambda matches where you set up your SQS queue, DynamoDB, and your parameter store!
- Make sure your lambda execution role has permissions to access the parameter store!
- Create a trigger on the lambda that uses the SQS queue
meetupgroups
. Set the batch size to 1.
This lambda retrieves all the events for a given location from the database and turns them into two iCal calendars: one with lots of separate events and a digest that has one all-day event per day. These are uploaded to S3. This will be triggered from another Step Function that we'll set up shortly.
- You need to create an S3 bucket to write to.
- Make sure the region in the lambda matches the one where you set up your DynamoDB table!
To trigger these lambdas to run, we're going to set up two Step Functions (state machines).
Create a State Machine called 'MUGicalNode' using the 'Author with Code Snippets' option, and for the definition paste in the Amazon States Language from sf-mugicalnode.txt
. You'll need to change the resource definitions to point your getcities
and getgroups
lambdas. It will call the getcities
lambda, and then loop through it calling getgroups
on each city.
Create a State Machine called 'MUGicalNode-Calendar' using the 'Author with Code Snippets' option, and for the definition paste in the Amazon States Language from sf-mugicalnodecal.txt
. You'll need to change the resource definitions to point your getcities
and makecalendar
lambdas. It will call the getcities
lambda, and then loop through it calling makecalendar
on each city.
To actually kick off the Step Functions, you need to create two CloudWatch Rules.
- Create a Rule called 'MUGicalNode' using the 'Schedule' option and enter
0 1 * * ? *
as the cron expression. This will run every day at 1am. Change the target to Step Functions state machine and select the 'MUGicalNode' state machine you created in the previous step. - Create a Rule called 'MUGicalNodeCalendar' using the 'Schedule' option and enter
00 23 * * ? *
as the cron expression. This will run every night at 11pm. Change the target to Step Functions state machine and select the 'MUGicalNode-Calendar' state machine you created in the previous step.
And that's it!
The code for this repository has been released into the public domain by Kristine Howard via the Unlicense.
Originally based on Open Austin's meetup-proxy-aws-lambda.
Created by @web-goddess.