New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import-Vaporshell causes ConvertFrom-Json to Fail on PS4.0 #15

Closed
RegEM opened this Issue Aug 29, 2017 · 45 comments

Comments

2 participants
@RegEM
Copy link

RegEM commented Aug 29, 2017

Hi great project.
I am stuck though running Import-Vaporshell though. I get this error, and don't know how to fix it.
Running this: $template = Import-Vaporshell .\awstemplate.json
Tried full path and in quotes etc, but haven't figured out the secret.
Please help me.

"ConvertFrom-Json : Invalid object passed in, ':' or '}' expected. (1): {
At line:36 char:24

  • $tempObj = $temp | ConvertFrom-Json -Verbose:$false
    
    • CategoryInfo : NotSpecified: (:) [ConvertFrom-Json], ArgumentException
    • FullyQualifiedErrorId : System.ArgumentException,Microsoft.PowerShell.Commands.ConvertFromJsonCommand"
@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

I'm glad you dig! From the error, it looks like the template JSON is malformed?

Import-Vaporshell does a few things:

  1. Gets the content of the JSON file at that path, then converts from JSON into a PSObject
  2. Adds various script methods to the object to allow adding/removing resources,outputs,parameters,etc.
  3. Adds the custom object type "Vaporshell.Template" for use during validation when exporting it.

It looks like it's hanging on step one there.

Try this and see if you get a similar error:

Get-Content .\awstemplate.json | ConvertFrom-Json

If you're getting the same thing, check out the template you're importing from to ensure is correctly formatted and not missing something like a closing brace or colon

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

Thx.
The templates I am trying to import, are exported from vaposhell.
I am validating too, "VERBOSE: The template was validated successfully!"
Here is one example. Not sure how to post json.
{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "CFN - tcam utility functions", "Transform": "AWS::Serverless-2016-10-31", "Resources": { "sometbl": { "Type": "AWS::Serverless::SimpleTable" }, "tcamUpdateMonthstable": { "Type": "AWS::Serverless::Function", "Properties": { "Handler": "handler.updateSomestable", "Runtime": "nodejs6.10", "CodeUri": "s3://somebucket/update_table.zip", "Policies": "AmazonDynamoDBReadOnlyAccess", "Environment": { "Variables": { "TABLE_NAME": { "Ref": "sometbl" } } } } } } }
I tried your suggestion. No improvement.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

Weird! Checking this out now!

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

I see this reference, but I'm not that conversant in PS.
https://stackoverflow.com/questions/18243707/powershell-convertfrom-json-error

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

If I wrap that JSON example in single quotes and pipe it directly to ConvertFrom-Json, it works fine for me.

'{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "CFN - tcam utility functions", "Transform": "AWS::Serverless-2016-10-31", "Resources": { "sometbl": { "Type": "AWS::Ser
verless::SimpleTable" }, "tcamUpdateMonthstable": { "Type": "AWS::Serverless::Function", "Properties": { "Handler": "handler.updateSomestable", "Runtime": "nodejs6.10", "CodeUri": "s3://so
mebucket/update_table.zip", "Policies": "AmazonDynamoDBReadOnlyAccess", "Environment": { "Variables": { "TABLE_NAME": { "Ref": "sometbl" } } } } } } }' | ConvertFrom-Json

AWSTemplateFormatVersion Description                  Transform                  Resources
------------------------ -----------                  ---------                  ---------
2010-09-09               CFN - tcam utility functions AWS::Serverless-2016-10-31 @{sometbl=; tcamUpdateMonthstable=}

What version of Powershell are you using? Easiest would be to paste the output of $PSVersionTable here

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

Name Value


PSVersion 4.0
WSManStackVersion 3.0
SerializationVersion 1.1.0.1
CLRVersion 4.0.30319.36366
BuildVersion 6.3.9600.17400
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0}
PSRemotingProtocolVersion 2.2

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

As you did, direct in quotes works for me too.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

Alright... Can you try using the full path instead of `.\awstemplate.json'? Checking if it's not resolving the path correctly

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

I tried already, but will again.

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

Tried,
PS C:\Users\rr\Documents\Vaporshell> Get-Content "C:\Users\rr\Documents\Vaporshell\template.json" | ConvertFrom-Json and without quotes. No luck.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

would you be able to attach that template.json here so I can check it out? if it contains anything sensitive, don't stress on it. There's something odd about the json itself though if it's not even working when piped into ConvertFrom-Json

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

So you know, the template is a bit more complicated, but does validate, and is accepted in quotes on the command line. Hence the different name.
I recently bought a new laptop. I will move to that, maybe later today, and confirm it is just a problem on the old one. Will probably be ok.
Nice to fix this issue, but I suggest not expending too much effort. Not unless you get other reporters.

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

Renamed with .txt, as couldn't post otherwise.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

Weird! it's working fine for me importing straight off that file:

I ♥ PS>Import-Vaporshell -Path "C:\Users\nferrell\Downloads\template.json.txt"

AWSTemplateFormatVersion Description                  Transform                  Resources
------------------------ -----------                  ---------                  ---------
2010-09-09               CFN - tcam utility functions AWS::Serverless-2016-10-31 @{tcamMonthstbl=; tcamDaystbl=; tcamHourstbl=; tcamUpdateMonthstable=}
@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

didn't see anything wrong with the file either when I checked it out in VSCode, not linting issues, etc... I'd suspect something local to that computer, interested in hearing about success / etc on your other one!

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

Well thanks a lot for checking. If you think of anything let me know. I've been limping this laptop along for a while. Time to move on...

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

last attempt: can you try getting rid of the pipeline and seeing if the following works for you on that PC?

ConvertFrom-Json -InputObject ([System.IO.File]::ReadAllText("C:\Users\rr\Documents\Vaporshell\template.json"))
@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

Yes. that works.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

Cool! Alright, I should have version 2.0 released by this weekend. I'll swap the way that Import-Vaporshell is using ConvertFrom-Json to that. It looks like pipeline issues with how I built the function.

For the record, I was testing in PS5.1 and 6.0.0-beta5, I did not have a PS4 machine to test on at the moment.

I appreciate submitting this! I'm going to keep it open as a bug and close it out once the new version is released. Cheers!

@scrthq scrthq self-assigned this Aug 29, 2017

@scrthq scrthq added the bug label Aug 29, 2017

@scrthq scrthq changed the title wrestling with Import-Vaporshell Import-Vaporshell causes ConvertFrom-Json to Fail on PS4.0 Aug 29, 2017

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 29, 2017

Awsome. Glad to help. I look forward to v2. Have a great day!

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 29, 2017

Cheers, you too!

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

Nate, Import-Vaporshell works on my new Win10 laptop.

Unfortunately having some other config issue. Fails on both validatingtemplate, and awscli didn't seem to install correctly. Not sure if powershellget is an issue. Need to setup the pc a bit. Will let you know if it is an issue. Prob. be a few days.

I'll add that I jumped over to the new laptop because of can't use/do transform error on the old one. Probably an old version of awscli or something.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 30, 2017

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

ok. seems likely I installed cli instead of sdk. should have time this morning to check, the vaporshell was latest, afaik.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 30, 2017

The SDK is actually bundled with the module now (v.1.2.3), so you shouldn't need to install anything extra =]... Send me the error you're getting if you can!

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

ok. that makes more sense to me. I was just installing node etc. (I needed to anyways).

I believe I have just had to re-install cfn-flip, so I'm not sure if my install permissions are allowing the install to be done correctly. Anyways, after getting rid of a cfn-flip error, I am back to the following: (Sorry I am not properly setup on the laptop, so this is just summarizing the error)

I get,

  1. "exception calling .ctor with 1 arg... "No region endpoint or serviceurl configured"
    At line 19 char 9, $req = ....
  2. some "Write-S3Object not recognized, I think that is on the Export
  3. some New-CFN-Stack not recognized
@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

New-CFNStack error is of course mine, as I am running from a ps1 script

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

The Writes3 appears to be from my script too. I have just installed AWSPowershell, but the setup is still not quite correct

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

I dunno, my credentials are not persisting properly on Win10.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 30, 2017

ah, I get ya! looks like you need to add your region to your credentials file.

You can set/update it with Set-VSCredentials:

NAME
    Set-VSCredentials

SYNTAX
    Set-VSCredentials [[-ProfileName] <string>] [-AccessKey <string>] [-SecretKey <string>] [-Region {APNortheast1 | APNortheast2 | APSouth1 | APSoutheast1 | APSoutheast2 | CACentral1 |
    CNNorth1 | EUCentral1 | EUWest1 | EUWest2 | SAEast1 | USEast1 | USEast2 | USGovCloudWest1 | USWest1 | USWest2}]  [<CommonParameters>]
@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

OK. I'll look at that. I did manage to eliminate error 3 by using Set-DefaultAWSRegion. I'll add that the command was different in earlier versions, I think.

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

Ran Set-VSCredentials, which seems to have fixed Vaporshell, as it now gets right through to the validation.

I still have my local PS errors. Doesn't find Write-S3Object for example, but can from command line. I'll figure it out.

Error 3 has changed to the same error I was getting on teh old laptop, New-CFNStack : Create Stack cannot be used with templates containg transforms"

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 30, 2017

For sure! The updates coming in v2 outside of small fixes are all AWS SDK wrappers, which the AWSPowerShell module already does; i.e. the following would work the same:

Set-AWSCredential -AccessKey AKIAIOSFODNN7EXAMPLE -SecretKey wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -StoreAs myCredentials

Set-VSCredentials -AccessKey AKIAIOSFODNN7EXAMPLE -SecretKey wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -ProfileName myCredentials

The main difference between AWSPowershell and Vaporshell is that you can also specify the -Region with Set-VSCredentials. Also, Vaporshell comes with both .Net45 and .NetCore SDK's and will load the appropriate assemblies on module import depending on the Powershell version, meaning your Vaporshell scripts would work whether you're running them in Powershell 4/5 on Windows or Powershell 6 on Ubuntu. You would need to import AWSPowershell.NetCore instead of AWSPowershell when working in CoreCLR.


For Serverless (or any template with Transforms), I believe you would need to create a Change Set, review, then execute it instead of deploying the stack straight away since it has to process the Transforms first: https://github.com/awslabs/serverless-application-model/blob/master/HOWTO.md#deploying-to-aws-cloudformation

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

Ok. Thanks for the great explanation. I believe I even understand most of it.
I was hoping it was a bit easier re: the transforms, as my starting point was your API backend example. I just tagged on a few elements. A deployment example or two would of course be fantastic.
I will go back to the old laptop now with more confidence. Too much going on to move to the new one right now anyways.

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

hmmm, I thought you just asked me to send you the error. Can do, if you still want it? Disregard, Github is acting weird on me.

@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

I changed my deployment from New-CFNStack type to "aws cloudformation deploy --template-file 'https://tcam-vapo-fn.s3.amazonaws.com/template.yaml' --stack-name 'CFNtcamUtilities' --capabilities CAPABILITY_IAM"
And it worked fine, creating the change set on the fly. It didn't quite finish properly, but I just hit return to get the success text.
Thanks for pointing me in the right direction.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 30, 2017

Awesome! With Vaporshell 2.0, you'd be able to do something similar to the following to create the change set then execute it:

New-VSChangeSet -TemplateUrl 'https://tcam-vapo-fn.s3.amazonaws.com/template.yaml' -Capabilities CAPABILITY_IAM -StackName CFNtcamUtilities

Invoke-VSChangeSetExecution -StackName CFNtcamUtilities
@RegEM

This comment has been minimized.

Copy link

RegEM commented Aug 30, 2017

cool. I'm just trying to knock off some super simple lambda functions and iot rules, and this should help me do that. I hope you have a lot of success with this.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Aug 30, 2017

Thank you!! Definitely a passion project right now, hopefully it becomes something essential to a lot of CFN gurus 🥇

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Sep 1, 2017

@RegEM -- Just noticed something odd about calling the SDK from Powershell 3/4 that doesn't happen in 5 that is likely causing your -ValidateTemplate flag to fail (and will also likely kick an error when you try to use Set-VSCredentials). Assuming the error you get looks a bit like this?

Method invocation failed because [Amazon.Runtime.StoredProfileAWSCredentials] does not contain a method named 'new'.

I've fixed it already in the new version and should hopefully have that out by this weekend!

@RegEM

This comment has been minimized.

Copy link

RegEM commented Sep 1, 2017

thanks.
I don't recall any fail on Set-VSCredentials, at all.
I think it was just that the command Initialize-AWSDefaults was replaced with Set-AWSCredential. So my shell commands were not authorized properly on the new box is all.
So I believe I could successfully run Vaporshell, including the validation, once I used either Set-AWS, or Set-VS.

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Sep 1, 2017

Gotcha! Well, it was at least something I encountered when testing in both PS v3 and v4 on a Win7 VM, so I felt it worth the mention. Regardless, I'm glad it was working fine for you and it'll be patched to work universally ASAP!

@RegEM

This comment has been minimized.

Copy link

RegEM commented Sep 1, 2017

Thanks Nate.
I'm looking forward to trying to import my stack that I have been working on, done with serverless. It's not super complicated, so I think it will be a nice test. I'm hoping to abandon serverless and do everything in vaporshell.
I'm starting over now and learning how to setup a proper dev environment. Or at least my guess on what will be good. So plenty to keep me busy while you fine tune v2..

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Sep 4, 2017

The only thing left now is to update documentation and deploy v2.0.0! Here's a GIF of a quick template build (S3 bucket only) to stack deployment (with New-VSStack) to stack tear down (with Remove-VSStack):

New-VSStack & Remove-VSStack in action

@scrthq

This comment has been minimized.

Copy link
Owner

scrthq commented Sep 16, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment